2023-10-21T03:47:46.382 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T03:47:46.383 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357 --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} --name rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi --verbose --owner scheduled_rraja@teuthology -- /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357/orig.config.yaml 2023-10-21T03:47:46.413 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T03:47:46.495 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357 branch: ajarr-wip-extra-debug-62994-oct-19 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7433357' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: ajarr-wip-extra-debug-62994-oct-19 ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: ajarr-wip-extra-debug-62994-oct-19 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1449 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab sleep_before_teardown: 0 suite: rbd:cli suite_branch: ajarr-wip-extra-debug-62994-oct-19 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAfZOxwwOSOg7SAAW1HKu994xk9gt1F1rsH50yetsOqqRSjPNPyrelDEyxFF3hzsEyq0o6huOWqCT9bTgCmORpc= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-20_15:15:00 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-21T03:47:46.496 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa; will attempt to use it 2023-10-21T03:47:46.496 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks 2023-10-21T03:47:46.497 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-21T03:47:46.499 INFO:teuthology.task.internal:Checking packages... 2023-10-21T03:47:46.523 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab' 2023-10-21T03:47:46.523 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-21T03:47:46.524 INFO:teuthology.packaging:ref: None 2023-10-21T03:47:46.524 INFO:teuthology.packaging:tag: None 2023-10-21T03:47:46.524 INFO:teuthology.packaging:branch: ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:47:46.525 INFO:teuthology.packaging:sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:47:46.525 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:47:46.757 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6744.g7f103c7c 2023-10-21T03:47:46.757 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-21T03:47:46.764 INFO:teuthology.task.internal:no buildpackages task found 2023-10-21T03:47:46.764 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-21T03:47:46.770 INFO:teuthology.task.internal:Saving configuration 2023-10-21T03:47:46.788 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-21T03:47:46.794 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T03:47:46.817 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:42:57.197671', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAfZOxwwOSOg7SAAW1HKu994xk9gt1F1rsH50yetsOqqRSjPNPyrelDEyxFF3hzsEyq0o6huOWqCT9bTgCmORpc='} 2023-10-21T03:47:46.818 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-21T03:47:46.824 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-21T03:47:46.824 INFO:teuthology.run_tasks:Running task console_log... 2023-10-21T03:47:46.867 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f822d53ea60>, signals=[15]) 2023-10-21T03:47:46.868 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-21T03:47:46.873 INFO:teuthology.task.internal:Opening connections... 2023-10-21T03:47:46.874 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T03:47:46.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:47:46.974 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-21T03:47:46.981 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-10-21T03:47:47.048 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-10-21T03:47:47.048 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-10-21T03:47:47.105 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2023-10-21T03:47:47.106 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2023-10-21T03:47:47.106 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2023-10-21T03:47:47.106 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2023-10-21T03:47:47.106 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2023-10-21T03:47:47.106 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2023-10-21T03:47:47.107 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-21T03:47:47.107 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-10-21T03:47:47.107 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2023-10-21T03:47:47.107 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-21T03:47:47.107 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2023-10-21T03:47:47.108 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-21T03:47:47.108 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-21T03:47:47.108 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-21T03:47:47.109 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-10-21T03:47:47.132 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-21T03:47:47.141 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-21T03:47:47.147 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-21T03:47:47.147 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-10-21T03:47:47.166 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-21T03:47:47.174 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-21T03:47:47.175 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-10-21T03:47:47.231 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-21T03:47:47.263 INFO:teuthology.run_tasks:Running task kernel... 2023-10-21T03:47:47.277 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-21T03:47:47.277 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-21T03:47:47.278 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-21T03:47:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T03:47:47.296 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-10-21T03:47:47.354 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-361.el9.x86_64 2023-10-21T03:47:47.354 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-361.el9.x86_64 2023-10-21T03:47:47.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2023-10-21T03:47:49.544 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 11 MB/s | 6.3 MB 00:00 2023-10-21T03:47:50.919 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2023-10-21T03:47:53.151 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 7.9 MB 00:01 2023-10-21T03:48:05.618 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 18 MB 00:11 2023-10-21T03:48:10.502 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.7 MB 00:01 2023-10-21T03:48:12.810 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 15 kB 00:01 2023-10-21T03:48:14.562 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 12 MB/s | 19 MB 00:01 2023-10-21T03:48:19.137 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2023-10-21T03:48:21.360 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-21T03:48:21.361 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-21T03:48:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:48:21.463 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:48:21.464 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:48:21.464 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:48:21.464 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:48:21.464 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-377.el9 CentOS-BaseOS 5.3 M 2023-10-21T03:48:21.465 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:48:21.465 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 5.14.0-377.el9 CentOS-BaseOS 20 M 2023-10-21T03:48:21.465 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 5.14.0-377.el9 CentOS-BaseOS 38 M 2023-10-21T03:48:21.465 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core x86_64 5.14.0-377.el9 CentOS-BaseOS 32 M 2023-10-21T03:48:21.466 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:48:21.466 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:48:21.466 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:48:21.466 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2023-10-21T03:48:21.466 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:48:21.467 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 95 M 2023-10-21T03:48:21.467 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 120 M 2023-10-21T03:48:21.468 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:48:22.038 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): kernel-5.14.0-377.el9.x86_64.rpm 9.3 MB/s | 5.3 MB 00:00 2023-10-21T03:48:22.538 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): kernel-core-5.14.0-377.el9.x86_64.rpm 19 MB/s | 20 MB 00:01 2023-10-21T03:48:23.605 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): kernel-modules-5.14.0-377.el9.x86_64.rpm 18 MB/s | 38 MB 00:02 2023-10-21T03:48:24.747 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): kernel-modules-core-5.14.0-377.el9.x86_6 12 MB/s | 32 MB 00:02 2023-10-21T03:48:24.750 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:48:24.751 INFO:teuthology.orchestra.run.smithi040.stdout:Total 29 MB/s | 95 MB 00:03 2023-10-21T03:48:24.751 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:48:25.180 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:48:25.180 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:48:26.189 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:48:26.189 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:48:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:48:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-core-5.14.0-377.el9.x86_64 1/4 2023-10-21T03:48:29.423 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-5.14.0-377.el9.x86_64 2/4 2023-10-21T03:48:32.269 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-377.el9.x86_64 2/4 2023-10-21T03:48:32.450 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-5.14.0-377.el9.x86_64 3/4 2023-10-21T03:48:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-377.el9.x86_64 3/4 2023-10-21T03:48:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-5.14.0-377.el9.x86_64 4/4 2023-10-21T03:48:42.457 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-core-5.14.0-377.el9.x86_64 4/4 2023-10-21T03:49:13.211 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-377.el9.x86_64 4/4 2023-10-21T03:49:13.256 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-377.el9.x86_64 4/4 2023-10-21T03:49:13.816 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-377.el9.x86_64 4/4 2023-10-21T03:49:13.817 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-377.el9.x86_64 1/4 2023-10-21T03:49:13.817 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-5.14.0-377.el9.x86_64 2/4 2023-10-21T03:49:13.817 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-5.14.0-377.el9.x86_64 3/4 2023-10-21T03:49:14.276 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-core-5.14.0-377.el9.x86_64 4/4 2023-10-21T03:49:14.277 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:49:14.277 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:49:14.277 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-377.el9.x86_64 2023-10-21T03:49:14.277 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core-5.14.0-377.el9.x86_64 2023-10-21T03:49:14.278 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-5.14.0-377.el9.x86_64 2023-10-21T03:49:14.278 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core-5.14.0-377.el9.x86_64 2023-10-21T03:49:14.278 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:49:14.278 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:49:14.734 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-10-21T03:49:14.859 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-377.el9.x86_64 2023-10-21T03:49:14.860 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-377.el9.x86_64 2023-10-21T03:49:14.860 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-377.el9.x86_64 2023-10-21T03:49:14.860 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-21T03:49:14.861 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-21T03:49:14.861 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-21T03:49:14.861 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2023-10-21T03:49:14.861 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-10-21T03:49:14.969 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-377.el9.x86_64 2023-10-21T03:49:14.970 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-377.el9.x86_64 2023-10-21T03:49:14.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2023-10-21T03:49:15.018 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2023-10-21T03:49:15.018 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2023-10-21T03:49:15.018 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2023-10-21T03:49:15.019 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2023-10-21T03:49:15.019 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2023-10-21T03:49:15.019 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-21T03:49:15.019 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2023-10-21T03:49:15.019 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2023-10-21T03:49:15.020 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2023-10-21T03:49:15.020 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-21T03:49:15.020 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-21T03:49:15.020 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-21T03:49:15.020 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-21T03:49:15.021 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2023-10-21T03:49:15.021 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2023-10-21T03:49:15.021 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-21T03:49:15.021 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2023-10-21T03:49:15.021 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2023-10-21T03:49:15.022 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:49:15.022 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-21T03:49:15.022 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-21T03:49:15.022 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-21T03:49:15.022 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2023-10-21T03:49:15.023 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:49:15.023 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2023-10-21T03:49:15.024 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-21T03:49:15.024 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 5.14.0-377.el9.x86_64 2023-10-21T03:49:15.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-21T03:49:15.623 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2023-10-21T03:49:16.988 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-21T03:49:17.022 INFO:teuthology.orchestra.run.smithi040.stderr:done 2023-10-21T03:49:17.024 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-10-21T03:49:17.039 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.0SikKsPARv 2023-10-21T03:49:17.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0SikKsPARv 2023-10-21T03:49:17.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.0SikKsPARv 2023-10-21T03:49:17.247 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.0SikKsPARv is 6KB 2023-10-21T03:49:17.296 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.0SikKsPARv 2023-10-21T03:49:17.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2023-10-21T03:49:17.378 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-21T03:49:17.379 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-21T03:49:17.379 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-21T03:49:17.379 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-377.el9.x86_64.conf 2023-10-21T03:49:17.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-377.el9.x86_64 2023-10-21T03:49:17.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2023-10-21T03:49:47.566 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-21T03:49:47.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-21T03:49:47.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:50:05.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-10-21T03:50:14.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-21T03:50:14.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:51:15.018 DEBUG:teuthology.orchestra.remote:timed out 2023-10-21T03:51:27.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-21T03:51:27.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:51:27.435 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-10-21T03:51:27.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-21T03:51:27.804 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-377.el9.x86_64"... 2023-10-21T03:51:27.805 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-10-21T03:51:27.820 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-377.el9.x86_64 2023-10-21T03:51:27.820 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-377.el9.x86_64 vs 5.14.0-377.el9.x86_64 2023-10-21T03:51:27.821 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-21T03:51:27.821 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-21T03:51:28.822 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-21T03:51:28.822 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-21T03:51:28.910 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2023-10-21T03:51:28.947 DEBUG:teuthology.parallel:result is None 2023-10-21T03:51:28.947 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-21T03:51:28.955 INFO:teuthology.task.internal:Creating test directory... 2023-10-21T03:51:28.955 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-21T03:51:28.974 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-21T03:51:28.980 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-21T03:51:28.987 INFO:teuthology.task.internal:Creating archive directory... 2023-10-21T03:51:28.988 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-21T03:51:29.046 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-21T03:51:29.053 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-21T03:51:29.054 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-21T03:51:29.124 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:51:29.142 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:51:29.144 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-21T03:51:29.151 INFO:teuthology.task.internal:Configuring sudo... 2023-10-21T03:51:29.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-21T03:51:29.219 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-21T03:51:29.228 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-21T03:51:29.228 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-21T03:51:29.276 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-21T03:51:29.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-21T03:51:29.425 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-21T03:51:29.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-21T03:51:29.516 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:51:29.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-21T03:51:29.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-10-21T03:51:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-21T03:51:29.861 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-21T03:51:29.869 INFO:teuthology.task.internal:Starting timer... 2023-10-21T03:51:29.870 INFO:teuthology.run_tasks:Running task pcp... 2023-10-21T03:51:29.879 INFO:teuthology.run_tasks:Running task selinux... 2023-10-21T03:51:29.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2023-10-21T03:51:29.937 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2023-10-21T03:51:29.939 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-21T03:51:29.940 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2023-10-21T03:51:29.977 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2023-10-21T03:51:29.978 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive'} 2023-10-21T03:51:29.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-21T03:51:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T03:51:30.049 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-21T03:51:30.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2023-10-21T03:51:30.127 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-21T03:51:30.136 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-21T03:51:30.137 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-21T03:51:30.155 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-21T03:51:30.157 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-21T03:57:05.098 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com')] 2023-10-21T03:57:05.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-21T03:57:05.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:57:05.202 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-10-21T03:57:05.268 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-21T03:57:05.269 INFO:teuthology.run_tasks:Running task clock... 2023-10-21T03:57:05.279 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-21T03:57:05.280 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-21T03:57:05.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T03:57:05.344 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-21T03:57:05.364 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-21T03:57:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2023-10-21T03:57:05.416 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2023-10-21T03:57:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-21T03:57:05.456 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-21T03:57:05.503 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2023-10-21T03:57:05.507 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-21T03:57:05.507 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-10-21T03:57:05.507 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-21T03:57:05.508 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-21T03:57:05.508 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-21T03:57:05.508 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-21T03:57:05.508 INFO:teuthology.run_tasks:Running task install... 2023-10-21T03:57:05.519 DEBUG:teuthology.task.install:project ceph 2023-10-21T03:57:05.520 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T03:57:05.520 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T03:57:05.520 INFO:teuthology.task.install:Using flavor: default 2023-10-21T03:57:05.526 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T03:57:05.527 INFO:teuthology.task.install:extra packages: [] 2023-10-21T03:57:05.528 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'tag': None, 'wait_for_package': False} 2023-10-21T03:57:05.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:57:05.681 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/ 2023-10-21T03:57:05.682 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6744.g7f103c7c 2023-10-21T03:57:05.792 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-21T03:57:05.793 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:57:05.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-21T03:57:05.825 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-21T03:57:05.826 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-21T03:57:05.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-21T03:57:05.986 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-21T03:57:06.015 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2023-10-21T03:57:06.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2023-10-21T03:57:06.275 INFO:teuthology.orchestra.run.smithi040.stdout:72 files removed 2023-10-21T03:57:06.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2023-10-21T03:57:07.098 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 170 kB/s | 78 kB 00:00 2023-10-21T03:57:07.524 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 25 kB/s | 9.4 kB 00:00 2023-10-21T03:57:07.933 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2023-10-21T03:57:08.318 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-10-21T03:57:09.711 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2023-10-21T03:57:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 7.9 MB 00:01 2023-10-21T03:57:13.586 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 13 MB/s | 18 MB 00:01 2023-10-21T03:57:17.779 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 5.7 MB 00:00 2023-10-21T03:57:19.579 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 15 kB 00:00 2023-10-21T03:57:22.514 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 19 MB 00:02 2023-10-21T03:57:27.206 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2023-10-21T03:57:29.741 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:57:29.746 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:57:29.746 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:57:29.746 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:57:29.747 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:57:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 8.7 M 2023-10-21T03:57:29.747 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2023-10-21T03:57:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 3.3 M 2023-10-21T03:57:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 3.0 M 2023-10-21T03:57:29.748 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:57:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options 2023-10-21T03:57:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-21T03:57:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 5.2 M 2023-10-21T03:57:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 19 M 2023-10-21T03:57:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 24 k 2023-10-21T03:57:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-21T03:57:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-21T03:57:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-21T03:57:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-21T03:57:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 716 k 2023-10-21T03:57:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-21T03:57:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-21T03:57:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-21T03:57:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 473 k 2023-10-21T03:57:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-21T03:57:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 4.8 M 2023-10-21T03:57:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-21T03:57:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-21T03:57:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-21T03:57:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse 2023-10-21T03:57:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 45 k 2023-10-21T03:57:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 127 k 2023-10-21T03:57:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 162 k 2023-10-21T03:57:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-21T03:57:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 320 k 2023-10-21T03:57:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 298 k 2023-10-21T03:57:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 99 k 2023-10-21T03:57:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-21T03:57:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-21T03:57:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:29.757 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:57:29.757 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:57:29.757 INFO:teuthology.orchestra.run.smithi040.stdout:Install 28 Packages 2023-10-21T03:57:29.758 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2023-10-21T03:57:29.758 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:29.759 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 55 M 2023-10-21T03:57:29.759 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:57:31.107 INFO:teuthology.orchestra.run.smithi040.stdout:(1/30): ceph-base-18.0.0-6744.g7f103c7c.el9.x86 7.2 MB/s | 5.2 MB 00:00 2023-10-21T03:57:31.149 INFO:teuthology.orchestra.run.smithi040.stdout:(2/30): ceph-selinux-18.0.0-6744.g7f103c7c.el9. 577 kB/s | 24 kB 00:00 2023-10-21T03:57:31.224 INFO:teuthology.orchestra.run.smithi040.stdout:(3/30): libcephfs2-18.0.0-6744.g7f103c7c.el9.x8 9.4 MB/s | 716 kB 00:00 2023-10-21T03:57:31.308 INFO:teuthology.orchestra.run.smithi040.stdout:(4/30): libradosstriper1-18.0.0-6744.g7f103c7c. 5.5 MB/s | 473 kB 00:00 2023-10-21T03:57:31.816 INFO:teuthology.orchestra.run.smithi040.stdout:(5/30): librgw2-18.0.0-6744.g7f103c7c.el9.x86_6 9.4 MB/s | 4.8 MB 00:00 2023-10-21T03:57:31.867 INFO:teuthology.orchestra.run.smithi040.stdout:(6/30): python3-ceph-argparse-18.0.0-6744.g7f10 903 kB/s | 45 kB 00:00 2023-10-21T03:57:31.942 INFO:teuthology.orchestra.run.smithi040.stdout:(7/30): python3-ceph-common-18.0.0-6744.g7f103c 1.7 MB/s | 127 kB 00:00 2023-10-21T03:57:32.009 INFO:teuthology.orchestra.run.smithi040.stdout:(8/30): python3-cephfs-18.0.0-6744.g7f103c7c.el 2.4 MB/s | 162 kB 00:00 2023-10-21T03:57:32.084 INFO:teuthology.orchestra.run.smithi040.stdout:(9/30): python3-rados-18.0.0-6744.g7f103c7c.el9 4.2 MB/s | 320 kB 00:00 2023-10-21T03:57:32.151 INFO:teuthology.orchestra.run.smithi040.stdout:(10/30): python3-rbd-18.0.0-6744.g7f103c7c.el9. 4.4 MB/s | 298 kB 00:00 2023-10-21T03:57:32.218 INFO:teuthology.orchestra.run.smithi040.stdout:(11/30): python3-rgw-18.0.0-6744.g7f103c7c.el9. 1.5 MB/s | 99 kB 00:00 2023-10-21T03:57:32.318 INFO:teuthology.orchestra.run.smithi040.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2023-10-21T03:57:32.360 INFO:teuthology.orchestra.run.smithi040.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2023-10-21T03:57:32.410 INFO:teuthology.orchestra.run.smithi040.stdout:(14/30): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-10-21T03:57:32.453 INFO:teuthology.orchestra.run.smithi040.stdout:(15/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2023-10-21T03:57:32.478 INFO:teuthology.orchestra.run.smithi040.stdout:(16/30): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2023-10-21T03:57:32.745 INFO:teuthology.orchestra.run.smithi040.stdout:(17/30): ceph-common-18.0.0-6744.g7f103c7c.el9. 8.0 MB/s | 19 MB 00:02 2023-10-21T03:57:32.845 INFO:teuthology.orchestra.run.smithi040.stdout:(18/30): ceph-radosgw-18.0.0-6744.g7f103c7c.el9 3.5 MB/s | 8.7 MB 00:02 2023-10-21T03:57:32.921 INFO:teuthology.orchestra.run.smithi040.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 632 kB/s | 111 kB 00:00 2023-10-21T03:57:32.979 INFO:teuthology.orchestra.run.smithi040.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.3 MB/s | 309 kB 00:00 2023-10-21T03:57:33.013 INFO:teuthology.orchestra.run.smithi040.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 734 kB/s | 24 kB 00:00 2023-10-21T03:57:33.046 INFO:teuthology.orchestra.run.smithi040.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2023-10-21T03:57:33.080 INFO:teuthology.orchestra.run.smithi040.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2023-10-21T03:57:33.147 INFO:teuthology.orchestra.run.smithi040.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 12 MB/s | 837 kB 00:00 2023-10-21T03:57:33.206 INFO:teuthology.orchestra.run.smithi040.stdout:(25/30): libarrow-9.0.0-9.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2023-10-21T03:57:33.231 INFO:teuthology.orchestra.run.smithi040.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2023-10-21T03:57:33.281 INFO:teuthology.orchestra.run.smithi040.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2023-10-21T03:57:33.406 INFO:teuthology.orchestra.run.smithi040.stdout:(28/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 49 kB/s | 45 kB 00:00 2023-10-21T03:57:33.590 INFO:teuthology.orchestra.run.smithi040.stdout:(29/30): librados2-18.0.0-6744.g7f103c7c.el9.x8 9.2 MB/s | 3.3 MB 00:00 2023-10-21T03:57:33.648 INFO:teuthology.orchestra.run.smithi040.stdout:(30/30): librbd1-18.0.0-6744.g7f103c7c.el9.x86_ 8.3 MB/s | 3.0 MB 00:00 2023-10-21T03:57:33.652 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:57:33.652 INFO:teuthology.orchestra.run.smithi040.stdout:Total 14 MB/s | 55 MB 00:03 2023-10-21T03:57:33.665 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:57:33.707 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:57:33.707 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:57:34.199 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:57:34.199 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:57:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:57:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-21T03:57:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-21T03:57:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-21T03:57:36.161 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-21T03:57:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:18.0.0-6744.g7f103c7c.el9.x86_64 5/32 2023-10-21T03:57:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:18.0.0-6744.g7f103c7c.el9.x86_64 5/32 2023-10-21T03:57:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:18.0.0-6744.g7f103c7c.el9.x86_64 6/32 2023-10-21T03:57:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:18.0.0-6744.g7f103c7c.el9.x86_64 7/32 2023-10-21T03:57:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:18.0.0-6744.g7f103c7c.el9.x86_64 7/32 2023-10-21T03:57:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-21T03:57:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:18.0.0-6744.g7f103c7c.el9. 9/32 2023-10-21T03:57:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:18.0.0-6744.g7f103c7c.el9.x86_64 10/32 2023-10-21T03:57:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:18.0.0-6744.g7f103c7c.el9.x86_6 11/32 2023-10-21T03:57:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6744.g7f103c7c.el9.x86_6 11/32 2023-10-21T03:57:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-21T03:57:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-21T03:57:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-21T03:57:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-21T03:57:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-21T03:57:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-21T03:57:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-21T03:57:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-21T03:57:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:18.0.0-6744.g7f103c7c.el9.x86_64 20/32 2023-10-21T03:57:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:18.0.0-6744.g7f103c7c.el9.x86_64 20/32 2023-10-21T03:57:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 21/32 2023-10-21T03:57:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-21T03:57:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-21T03:57:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:18.0.0-6744.g7f103c7c.el9.x86_64 24/32 2023-10-21T03:57:38.043 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:18.0.0-6744.g7f103c7c.el9.x86_64 24/32 2023-10-21T03:57:38.122 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 25/32 2023-10-21T03:57:38.148 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:18.0.0-6744.g7f103c7c.el9.x8 26/32 2023-10-21T03:57:40.397 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-6744.g7f103c7c.el9.x86_64 27/32 2023-10-21T03:57:40.437 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:18.0.0-6744.g7f103c7c.el9.x86_64 27/32 2023-10-21T03:57:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-6744.g7f103c7c.el9.x86_64 27/32 2023-10-21T03:57:41.059 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:18.0.0-6744.g7f103c7c.el9.x86_64 28/32 2023-10-21T03:57:41.145 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:18.0.0-6744.g7f103c7c.el9.x86_64 28/32 2023-10-21T03:57:41.146 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-21T03:57:41.146 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-21T03:57:41.146 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:41.252 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:18.0.0-6744.g7f103c7c.el9.x86_64 29/32 2023-10-21T03:57:51.418 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6744.g7f103c7c.el9.x86_64 29/32 2023-10-21T03:57:51.418 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2023-10-21T03:57:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2023-10-21T03:57:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2023-10-21T03:57:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2023-10-21T03:57:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2023-10-21T03:57:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2023-10-21T03:57:51.420 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2023-10-21T03:57:51.420 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:52.279 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 30/32 2023-10-21T03:57:52.305 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 30/32 2023-10-21T03:57:52.305 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:57:52.305 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-21T03:57:52.306 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-21T03:57:52.306 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-21T03:57:52.306 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:52.307 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-21T03:57:52.382 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-21T03:57:52.382 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-21T03:57:53.528 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-21T03:57:53.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/32 2023-10-21T03:57:53.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:18.0.0-6744.g7f103c7c.el9.x86_64 2/32 2023-10-21T03:57:53.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 3/32 2023-10-21T03:57:53.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:18.0.0-6744.g7f103c7c.el9.x86_64 4/32 2023-10-21T03:57:53.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:18.0.0-6744.g7f103c7c.el9.x86_64 5/32 2023-10-21T03:57:53.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:18.0.0-6744.g7f103c7c.el9.x86_6 6/32 2023-10-21T03:57:53.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:18.0.0-6744.g7f103c7c.el9.x86_64 7/32 2023-10-21T03:57:53.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6744.g7f103c7c.el9. 8/32 2023-10-21T03:57:53.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:18.0.0-6744.g7f103c7c.el9.x8 9/32 2023-10-21T03:57:53.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:18.0.0-6744.g7f103c7c.el9.x86_64 10/32 2023-10-21T03:57:53.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:18.0.0-6744.g7f103c7c.el9.x86_64 11/32 2023-10-21T03:57:53.531 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 12/32 2023-10-21T03:57:53.531 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 13/32 2023-10-21T03:57:53.531 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-21T03:57:53.531 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-21T03:57:53.531 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-21T03:57:53.532 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-21T03:57:53.532 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-21T03:57:53.532 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-21T03:57:53.532 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-21T03:57:53.533 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-21T03:57:53.533 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-21T03:57:53.533 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-21T03:57:53.533 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-21T03:57:53.534 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-21T03:57:53.534 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-21T03:57:53.534 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-21T03:57:53.534 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-21T03:57:53.534 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:18.0.0-6744.g7f103c7c.el9.x86_64 29/32 2023-10-21T03:57:53.535 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-21T03:57:53.535 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:18.0.0-6744.g7f103c7c.el9.x86_64 31/32 2023-10-21T03:57:54.615 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-21T03:57:54.616 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:54.616 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2023-10-21T03:57:54.616 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.617 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.617 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:57:54.617 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-21T03:57:54.617 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.617 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.617 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.618 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.618 INFO:teuthology.orchestra.run.smithi040.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-21T03:57:54.618 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-21T03:57:54.618 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-21T03:57:54.618 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-21T03:57:54.619 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.619 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-21T03:57:54.619 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-21T03:57:54.620 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-21T03:57:54.620 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.620 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-21T03:57:54.621 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.621 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-21T03:57:54.622 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-21T03:57:54.622 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-21T03:57:54.622 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.623 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.623 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.623 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-21T03:57:54.623 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.623 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.624 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:57:54.624 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2023-10-21T03:57:54.624 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-21T03:57:54.624 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:54.624 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:57:54.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2023-10-21T03:57:55.543 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:28 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:57:55.687 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:57:55.689 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:57:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:57:55.689 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:57:55.689 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:57:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 41 M 2023-10-21T03:57:55.690 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:57:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-21T03:57:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 164 k 2023-10-21T03:57:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-21T03:57:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-21T03:57:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-21T03:57:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:55.692 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:57:55.692 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:57:55.692 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2023-10-21T03:57:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:57:55.693 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 42 M 2023-10-21T03:57:55.693 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 172 M 2023-10-21T03:57:55.693 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:57:56.002 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-10-21T03:57:56.061 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-10-21T03:57:56.111 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-10-21T03:57:56.245 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): libcephsqlite-18.0.0-6744.g7f103c7c.el9. 415 kB/s | 164 kB 00:00 2023-10-21T03:57:56.362 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 747 kB/s | 187 kB 00:00 2023-10-21T03:57:59.178 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): ceph-test-18.0.0-6744.g7f103c7c.el9.x86_ 12 MB/s | 41 MB 00:03 2023-10-21T03:57:59.181 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:57:59.182 INFO:teuthology.orchestra.run.smithi040.stdout:Total 12 MB/s | 42 MB 00:03 2023-10-21T03:57:59.196 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:57:59.220 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:57:59.220 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:57:59.597 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:57:59.598 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:00.157 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:00.247 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-21T03:58:00.288 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-21T03:58:00.320 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-21T03:58:00.362 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-21T03:58:00.389 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:18.0.0-6744.g7f103c7c.el9.x86_64 5/6 2023-10-21T03:58:04.428 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6744.g7f103c7c.el9.x86_64 5/6 2023-10-21T03:58:04.508 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:18.0.0-6744.g7f103c7c.el9.x86_64 6/6 2023-10-21T03:58:06.523 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:18.0.0-6744.g7f103c7c.el9.x86_64 6/6 2023-10-21T03:58:06.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/6 2023-10-21T03:58:06.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:18.0.0-6744.g7f103c7c.el9.x86_64 2/6 2023-10-21T03:58:06.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-21T03:58:06.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-21T03:58:06.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-21T03:58:06.927 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-21T03:58:06.927 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:06.927 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:06.928 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:06.928 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-15.el9.x86_64 2023-10-21T03:58:06.928 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:06.928 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-21T03:58:06.928 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-21T03:58:06.928 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-21T03:58:06.929 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:06.929 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:07.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2023-10-21T03:58:07.754 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:40 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:07.912 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:07.914 INFO:teuthology.orchestra.run.smithi040.stdout:========================================================================================= 2023-10-21T03:58:07.915 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:07.915 INFO:teuthology.orchestra.run.smithi040.stdout:========================================================================================= 2023-10-21T03:58:07.915 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:07.915 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 6.5 k 2023-10-21T03:58:07.915 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:58:07.916 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 2.1 M 2023-10-21T03:58:07.916 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 1.5 M 2023-10-21T03:58:07.916 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 243 k 2023-10-21T03:58:07.916 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 4.4 M 2023-10-21T03:58:07.917 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 18 M 2023-10-21T03:58:07.917 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-21T03:58:07.917 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-21T03:58:07.917 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-21T03:58:07.917 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-21T03:58:07.918 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-21T03:58:07.918 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-21T03:58:07.918 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-21T03:58:07.918 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-21T03:58:07.918 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-21T03:58:07.919 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-21T03:58:07.919 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-21T03:58:07.919 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-21T03:58:07.919 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-21T03:58:07.919 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-21T03:58:07.920 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-21T03:58:07.920 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-21T03:58:07.920 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-21T03:58:07.920 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-21T03:58:07.920 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-21T03:58:07.921 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-21T03:58:07.921 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-21T03:58:07.921 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-21T03:58:07.921 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-21T03:58:07.921 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-21T03:58:07.922 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-21T03:58:07.922 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-21T03:58:07.922 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-21T03:58:07.922 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:07.923 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:07.923 INFO:teuthology.orchestra.run.smithi040.stdout:========================================================================================= 2023-10-21T03:58:07.923 INFO:teuthology.orchestra.run.smithi040.stdout:Install 33 Packages 2023-10-21T03:58:07.923 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:07.924 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 30 M 2023-10-21T03:58:07.924 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 110 M 2023-10-21T03:58:07.924 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:08.439 INFO:teuthology.orchestra.run.smithi040.stdout:(1/33): ceph-18.0.0-6744.g7f103c7c.el9.x86_64.r 33 kB/s | 6.5 kB 00:00 2023-10-21T03:58:08.606 INFO:teuthology.orchestra.run.smithi040.stdout:(2/33): ceph-mds-18.0.0-6744.g7f103c7c.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2023-10-21T03:58:09.042 INFO:teuthology.orchestra.run.smithi040.stdout:(3/33): ceph-mon-18.0.0-6744.g7f103c7c.el9.x86_ 7.3 MB/s | 4.4 MB 00:00 2023-10-21T03:58:09.118 INFO:teuthology.orchestra.run.smithi040.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6744.g7f10 3.2 MB/s | 243 kB 00:00 2023-10-21T03:58:09.216 INFO:teuthology.orchestra.run.smithi040.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-10-21T03:58:09.249 INFO:teuthology.orchestra.run.smithi040.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2023-10-21T03:58:09.283 INFO:teuthology.orchestra.run.smithi040.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-21T03:58:09.326 INFO:teuthology.orchestra.run.smithi040.stdout:(8/33): ceph-mgr-18.0.0-6744.g7f103c7c.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-10-21T03:58:09.350 INFO:teuthology.orchestra.run.smithi040.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2023-10-21T03:58:09.384 INFO:teuthology.orchestra.run.smithi040.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2023-10-21T03:58:09.409 INFO:teuthology.orchestra.run.smithi040.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2023-10-21T03:58:09.434 INFO:teuthology.orchestra.run.smithi040.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-21T03:58:09.460 INFO:teuthology.orchestra.run.smithi040.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-10-21T03:58:09.644 INFO:teuthology.orchestra.run.smithi040.stdout:(14/33): ceph-osd-18.0.0-6744.g7f103c7c.el9.x86 17 MB/s | 18 MB 00:01 2023-10-21T03:58:09.718 INFO:teuthology.orchestra.run.smithi040.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 197 kB/s | 51 kB 00:00 2023-10-21T03:58:09.745 INFO:teuthology.orchestra.run.smithi040.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 436 kB/s | 43 kB 00:00 2023-10-21T03:58:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2023-10-21T03:58:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2023-10-21T03:58:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2023-10-21T03:58:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-10-21T03:58:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2023-10-21T03:58:09.954 INFO:teuthology.orchestra.run.smithi040.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2023-10-21T03:58:09.979 INFO:teuthology.orchestra.run.smithi040.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2023-10-21T03:58:10.004 INFO:teuthology.orchestra.run.smithi040.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2023-10-21T03:58:10.046 INFO:teuthology.orchestra.run.smithi040.stdout:(25/33): python3-cryptography-36.0.1-4.el9.x86_ 2.0 MB/s | 1.2 MB 00:00 2023-10-21T03:58:10.080 INFO:teuthology.orchestra.run.smithi040.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 785 kB/s | 79 kB 00:00 2023-10-21T03:58:10.105 INFO:teuthology.orchestra.run.smithi040.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2023-10-21T03:58:10.130 INFO:teuthology.orchestra.run.smithi040.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2023-10-21T03:58:10.156 INFO:teuthology.orchestra.run.smithi040.stdout:(29/33): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2023-10-21T03:58:10.181 INFO:teuthology.orchestra.run.smithi040.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2023-10-21T03:58:10.206 INFO:teuthology.orchestra.run.smithi040.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2023-10-21T03:58:10.231 INFO:teuthology.orchestra.run.smithi040.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2023-10-21T03:58:10.265 INFO:teuthology.orchestra.run.smithi040.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2023-10-21T03:58:10.267 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:10.268 INFO:teuthology.orchestra.run.smithi040.stdout:Total 13 MB/s | 30 MB 00:02 2023-10-21T03:58:10.302 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:10.347 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:10.348 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:12.004 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-21T03:58:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-21T03:58:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-21T03:58:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-21T03:58:12.298 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-21T03:58:12.345 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-21T03:58:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-21T03:58:12.470 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-21T03:58:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-21T03:58:12.568 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-21T03:58:12.627 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-21T03:58:12.696 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-21T03:58:12.895 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-21T03:58:12.941 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-21T03:58:12.979 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-21T03:58:13.014 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-21T03:58:13.043 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-21T03:58:13.121 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-21T03:58:13.154 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-21T03:58:13.187 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-21T03:58:13.229 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-21T03:58:13.263 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-21T03:58:13.356 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-21T03:58:13.395 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-21T03:58:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-21T03:58:13.611 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-21T03:58:13.740 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-21T03:58:13.896 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6744.g7f103c7c.el9. 28/33 2023-10-21T03:58:13.923 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:18.0.0-6744.g7f103c7c.el9.x86_64 29/33 2023-10-21T03:58:13.950 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6744.g7f103c7c.el9.x86_64 29/33 2023-10-21T03:58:13.950 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:58:13.950 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-21T03:58:13.950 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-21T03:58:13.951 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-21T03:58:13.951 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:15.559 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:18.0.0-6744.g7f103c7c.el9.x86_64 30/33 2023-10-21T03:58:15.588 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:18.0.0-6744.g7f103c7c.el9.x86_64 30/33 2023-10-21T03:58:15.588 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:58:15.588 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-21T03:58:15.588 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-21T03:58:15.589 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-21T03:58:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:16.051 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:18.0.0-6744.g7f103c7c.el9.x86_64 31/33 2023-10-21T03:58:16.075 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:18.0.0-6744.g7f103c7c.el9.x86_64 31/33 2023-10-21T03:58:16.075 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:58:16.075 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-21T03:58:16.076 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-21T03:58:16.076 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-21T03:58:16.076 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:16.285 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:18.0.0-6744.g7f103c7c.el9.x86_64 32/33 2023-10-21T03:58:16.309 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:18.0.0-6744.g7f103c7c.el9.x86_64 32/33 2023-10-21T03:58:16.309 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:58:16.310 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-21T03:58:16.310 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-21T03:58:16.310 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-21T03:58:16.310 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:16.377 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:18.0.0-6744.g7f103c7c.el9.x86_64 33/33 2023-10-21T03:58:18.202 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:18.0.0-6744.g7f103c7c.el9.x86_64 33/33 2023-10-21T03:58:18.202 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/33 2023-10-21T03:58:18.202 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:18.0.0-6744.g7f103c7c.el9.x86_64 2/33 2023-10-21T03:58:18.202 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:18.0.0-6744.g7f103c7c.el9.x86_64 3/33 2023-10-21T03:58:18.203 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:18.0.0-6744.g7f103c7c.el9.x86_64 4/33 2023-10-21T03:58:18.203 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:18.0.0-6744.g7f103c7c.el9.x86_64 5/33 2023-10-21T03:58:18.203 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6744.g7f103c7c.el9. 6/33 2023-10-21T03:58:18.203 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-21T03:58:18.203 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-21T03:58:18.204 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-21T03:58:18.204 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-21T03:58:18.204 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-21T03:58:18.204 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-21T03:58:18.205 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-21T03:58:18.205 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-21T03:58:18.205 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-21T03:58:18.205 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-21T03:58:18.205 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-21T03:58:18.206 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-21T03:58:18.206 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-21T03:58:18.206 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-21T03:58:18.206 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-21T03:58:18.207 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-21T03:58:18.207 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-21T03:58:18.207 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-21T03:58:18.207 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-21T03:58:18.208 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-21T03:58:18.209 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-21T03:58:18.209 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-21T03:58:18.209 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-21T03:58:18.209 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-21T03:58:18.209 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-21T03:58:18.210 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-21T03:58:19.284 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-21T03:58:19.284 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:19.284 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:19.284 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:19.285 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:19.285 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:19.285 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:19.285 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:19.285 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-21T03:58:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-21T03:58:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-21T03:58:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-21T03:58:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-21T03:58:19.287 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-21T03:58:19.287 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-21T03:58:19.287 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-21T03:58:19.287 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-21T03:58:19.287 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-21T03:58:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-21T03:58:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-21T03:58:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-21T03:58:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-21T03:58:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-21T03:58:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2023-10-21T03:58:19.289 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-21T03:58:19.289 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-21T03:58:19.289 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-21T03:58:19.290 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-21T03:58:19.290 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-21T03:58:19.290 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-21T03:58:19.290 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-21T03:58:19.291 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-21T03:58:19.291 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-21T03:58:19.291 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-21T03:58:19.291 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-21T03:58:19.291 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:19.292 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:19.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2023-10-21T03:58:20.184 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:20.276 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:58:20.327 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:20.328 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:58:20.328 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:20.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install cephadm 2023-10-21T03:58:20.968 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:21.113 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:21.114 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:21.115 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 269 k 2023-10-21T03:58:21.115 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:21.115 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:21.115 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:21.115 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:58:21.116 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:21.116 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 269 k 2023-10-21T03:58:21.116 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 264 k 2023-10-21T03:58:21.116 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:21.350 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm-18.0.0-6744.g7f103c7c.el9.noarch.rpm 1.1 MB/s | 269 kB 00:00 2023-10-21T03:58:21.351 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:21.351 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.1 MB/s | 269 kB 00:00 2023-10-21T03:58:21.351 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:21.357 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:21.357 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:21.360 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:21.360 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:21.885 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:21.974 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:22.384 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:22.701 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:22.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:22.701 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:22.702 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:22.702 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:22.702 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:22.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache 2023-10-21T03:58:23.489 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:23.633 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:23.634 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:23.634 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-21T03:58:23.634 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:23.634 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:23.634 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 141 k 2023-10-21T03:58:23.635 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:58:23.635 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 141 k 2023-10-21T03:58:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 395 k 2023-10-21T03:58:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:23.948 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-18.0.0-6744.g7f103c 450 kB/s | 141 kB 00:00 2023-10-21T03:58:23.948 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:23.949 INFO:teuthology.orchestra.run.smithi040.stdout:Total 448 kB/s | 141 kB 00:00 2023-10-21T03:58:23.949 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:23.957 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:23.957 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:24.012 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:24.013 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:24.151 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:24.207 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6744.g7f103c7c. 1/1 2023-10-21T03:58:24.232 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6744.g7f103c7c. 1/1 2023-10-21T03:58:24.233 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:58:24.233 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-21T03:58:24.233 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:25.202 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6744.g7f103c7c. 1/1 2023-10-21T03:58:25.202 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:25.203 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:25.203 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:25.203 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:25.203 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:25.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2023-10-21T03:58:26.037 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:26.129 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:58:26.181 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:26.182 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:58:26.182 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:26.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2023-10-21T03:58:26.819 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:26.965 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:26.966 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:26.966 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:26.966 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:26.967 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:26.967 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 1.8 M 2023-10-21T03:58:26.967 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:58:26.967 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 23 k 2023-10-21T03:58:26.967 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 15 k 2023-10-21T03:58:26.968 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-21T03:58:26.968 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-21T03:58:26.968 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-21T03:58:26.968 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2023-10-21T03:58:26.968 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-21T03:58:26.969 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:26.969 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:26.969 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:26.969 INFO:teuthology.orchestra.run.smithi040.stdout:Install 7 Packages 2023-10-21T03:58:26.969 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:26.970 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.1 M 2023-10-21T03:58:26.970 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 8.6 M 2023-10-21T03:58:26.971 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:27.310 INFO:teuthology.orchestra.run.smithi040.stdout:(1/7): ceph-prometheus-alerts-18.0.0-6744.g7f10 60 kB/s | 15 kB 00:00 2023-10-21T03:58:27.327 INFO:teuthology.orchestra.run.smithi040.stdout:(2/7): ceph-grafana-dashboards-18.0.0-6744.g7f1 86 kB/s | 23 kB 00:00 2023-10-21T03:58:27.402 INFO:teuthology.orchestra.run.smithi040.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 98 kB/s | 9.0 kB 00:00 2023-10-21T03:58:27.436 INFO:teuthology.orchestra.run.smithi040.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 376 kB/s | 41 kB 00:00 2023-10-21T03:58:27.470 INFO:teuthology.orchestra.run.smithi040.stdout:(5/7): ceph-mgr-dashboard-18.0.0-6744.g7f103c7c 4.3 MB/s | 1.8 MB 00:00 2023-10-21T03:58:27.486 INFO:teuthology.orchestra.run.smithi040.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 367 kB/s | 31 kB 00:00 2023-10-21T03:58:27.536 INFO:teuthology.orchestra.run.smithi040.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2023-10-21T03:58:27.538 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:27.539 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.6 MB/s | 2.1 MB 00:00 2023-10-21T03:58:27.539 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:27.547 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:27.548 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:27.618 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:27.618 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:27.918 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:27.980 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-21T03:58:28.014 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-21T03:58:28.062 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-21T03:58:28.101 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-21T03:58:28.129 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6744.g7f103c7c.el9.n 5/7 2023-10-21T03:58:28.378 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6744.g7f103c7c.el9. 6/7 2023-10-21T03:58:28.407 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6744.g7f103c7c.el9.noarc 7/7 2023-10-21T03:58:28.938 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6744.g7f103c7c.el9.noarc 7/7 2023-10-21T03:58:28.939 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6744.g7f103c7c.el9. 1/7 2023-10-21T03:58:28.939 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6744.g7f103c7c.el9.noarc 2/7 2023-10-21T03:58:28.939 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6744.g7f103c7c.el9.n 3/7 2023-10-21T03:58:28.939 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-21T03:58:28.940 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-21T03:58:28.940 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-21T03:58:29.250 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-21T03:58:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:29.251 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-21T03:58:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-21T03:58:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-21T03:58:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-21T03:58:29.253 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:29.253 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:29.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-21T03:58:30.052 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:03 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:30.198 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:30.199 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2023-10-21T03:58:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:30.199 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2023-10-21T03:58:30.199 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 7.4 M 2023-10-21T03:58:30.200 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:58:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-21T03:58:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-21T03:58:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-21T03:58:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-21T03:58:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-21T03:58:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-21T03:58:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-21T03:58:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-21T03:58:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-21T03:58:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-21T03:58:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-21T03:58:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:30.202 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:30.203 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2023-10-21T03:58:30.203 INFO:teuthology.orchestra.run.smithi040.stdout:Install 12 Packages 2023-10-21T03:58:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:30.204 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 37 M 2023-10-21T03:58:30.204 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 209 M 2023-10-21T03:58:30.204 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:30.335 INFO:teuthology.orchestra.run.smithi040.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2023-10-21T03:58:30.360 INFO:teuthology.orchestra.run.smithi040.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 713 kB/s | 17 kB 00:00 2023-10-21T03:58:30.385 INFO:teuthology.orchestra.run.smithi040.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2023-10-21T03:58:30.469 INFO:teuthology.orchestra.run.smithi040.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-10-21T03:58:30.578 INFO:teuthology.orchestra.run.smithi040.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-10-21T03:58:30.603 INFO:teuthology.orchestra.run.smithi040.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 205 kB 00:00 2023-10-21T03:58:30.695 INFO:teuthology.orchestra.run.smithi040.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 4.8 MB 00:00 2023-10-21T03:58:30.720 INFO:teuthology.orchestra.run.smithi040.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-10-21T03:58:30.837 INFO:teuthology.orchestra.run.smithi040.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 6.9 MB/s | 803 kB 00:00 2023-10-21T03:58:31.038 INFO:teuthology.orchestra.run.smithi040.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2023-10-21T03:58:31.063 INFO:teuthology.orchestra.run.smithi040.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 876 kB/s | 195 kB 00:00 2023-10-21T03:58:32.021 INFO:teuthology.orchestra.run.smithi040.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 4.1 MB/s | 7.4 MB 00:01 2023-10-21T03:58:32.024 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:32.024 INFO:teuthology.orchestra.run.smithi040.stdout:Total 20 MB/s | 37 MB 00:01 2023-10-21T03:58:32.025 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:32.051 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:32.051 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:32.527 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:32.528 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:33.319 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-21T03:58:33.350 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-21T03:58:33.448 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-21T03:58:33.495 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-21T03:58:33.690 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-21T03:58:33.727 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-21T03:58:33.814 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-21T03:58:34.496 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-21T03:58:34.619 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-21T03:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-21T03:58:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-21T03:58:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6744.g7f103 12/12 2023-10-21T03:58:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6744.g7f103 12/12 2023-10-21T03:58:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6744.g7f103 1/12 2023-10-21T03:58:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-21T03:58:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-21T03:58:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-21T03:58:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-21T03:58:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-21T03:58:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-21T03:58:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-21T03:58:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-21T03:58:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-21T03:58:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-21T03:58:39.988 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-21T03:58:39.988 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:39.988 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:39.989 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:39.989 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-21T03:58:39.989 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-21T03:58:39.989 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-21T03:58:39.990 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-21T03:58:39.990 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-21T03:58:39.990 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-21T03:58:39.991 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-21T03:58:39.991 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-21T03:58:39.991 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-21T03:58:39.991 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-21T03:58:39.992 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-21T03:58:39.992 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:39.993 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:40.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2023-10-21T03:58:40.911 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:13 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:41.059 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:41.060 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-21T03:58:41.060 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:41.060 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-21T03:58:41.060 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:41.061 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 47 k 2023-10-21T03:58:41.061 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:58:41.061 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-21T03:58:41.061 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-21T03:58:41.061 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-21T03:58:41.062 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-21T03:58:41.062 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-21T03:58:41.062 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-21T03:58:41.062 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-21T03:58:41.062 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-21T03:58:41.063 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-21T03:58:41.063 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-21T03:58:41.063 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-21T03:58:41.063 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-21T03:58:41.063 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:41.064 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:41.064 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-21T03:58:41.064 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2023-10-21T03:58:41.064 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:41.065 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.0 M 2023-10-21T03:58:41.065 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 26 M 2023-10-21T03:58:41.065 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:41.368 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2023-10-21T03:58:41.394 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2023-10-21T03:58:41.444 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-10-21T03:58:41.469 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-21T03:58:41.494 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-21T03:58:41.520 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): ceph-mgr-rook-18.0.0-6744.g7f103c7c.el9 163 kB/s | 47 kB 00:00 2023-10-21T03:58:41.545 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2023-10-21T03:58:41.781 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2023-10-21T03:58:41.804 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2023-10-21T03:58:41.941 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2023-10-21T03:58:41.979 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 454 kB/s | 197 kB 00:00 2023-10-21T03:58:42.046 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 827 kB/s | 90 kB 00:00 2023-10-21T03:58:42.113 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2023-10-21T03:58:42.117 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:42.118 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2023-10-21T03:58:42.118 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:42.132 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:42.132 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:42.239 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:42.239 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:42.776 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:42.952 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-21T03:58:43.015 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-21T03:58:43.072 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-21T03:58:43.107 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-21T03:58:43.158 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-21T03:58:43.254 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-21T03:58:43.374 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-21T03:58:43.427 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-21T03:58:44.187 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-21T03:58:44.230 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-21T03:58:44.264 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-21T03:58:44.310 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-21T03:58:44.334 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:18.0.0-6744.g7f103c7c.el9.noarch 13/13 2023-10-21T03:58:45.087 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6744.g7f103c7c.el9.noarch 13/13 2023-10-21T03:58:45.087 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6744.g7f103c7c.el9.noarch 1/13 2023-10-21T03:58:45.088 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-21T03:58:45.088 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-21T03:58:45.088 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-21T03:58:45.088 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-21T03:58:45.088 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-21T03:58:45.089 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-21T03:58:45.089 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-21T03:58:45.089 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-21T03:58:45.089 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-21T03:58:45.089 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-21T03:58:45.090 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-21T03:58:45.492 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-21T03:58:45.492 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:45.492 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:45.492 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-21T03:58:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-21T03:58:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-21T03:58:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-21T03:58:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-21T03:58:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-21T03:58:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-21T03:58:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-21T03:58:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-21T03:58:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-21T03:58:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-21T03:58:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-21T03:58:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:45.495 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:45.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-cephadm 2023-10-21T03:58:46.289 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:19 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:46.435 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:46.435 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-21T03:58:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:46.436 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-21T03:58:46.436 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 132 k 2023-10-21T03:58:46.436 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:58:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-21T03:58:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-21T03:58:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-21T03:58:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-21T03:58:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-21T03:58:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:46.438 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:46.438 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-21T03:58:46.438 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2023-10-21T03:58:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:46.439 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 6.8 M 2023-10-21T03:58:46.439 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 33 M 2023-10-21T03:58:46.439 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:46.765 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-10-21T03:58:46.890 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-10-21T03:58:46.991 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6744.g7f103c7c.e 345 kB/s | 132 kB 00:00 2023-10-21T03:58:47.233 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2023-10-21T03:58:47.308 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2023-10-21T03:58:47.367 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 914 kB/s | 548 kB 00:00 2023-10-21T03:58:47.368 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:47.368 INFO:teuthology.orchestra.run.smithi040.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2023-10-21T03:58:47.369 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:47.379 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:47.380 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:47.475 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:47.476 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:47.778 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:47.962 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-21T03:58:48.027 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-21T03:58:48.581 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-21T03:58:48.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-21T03:58:48.742 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-21T03:58:48.772 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 6/6 2023-10-21T03:58:49.568 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 6/6 2023-10-21T03:58:49.569 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 1/6 2023-10-21T03:58:49.569 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-21T03:58:49.569 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-21T03:58:49.569 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-21T03:58:49.570 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-21T03:58:49.892 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-21T03:58:49.892 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:49.892 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:49.893 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:49.893 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-21T03:58:49.893 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-21T03:58:49.893 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-21T03:58:49.893 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-21T03:58:49.894 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-21T03:58:49.894 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:49.894 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:50.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2023-10-21T03:58:50.725 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:23 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:50.870 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:50.871 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:50.871 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:50.871 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:50.871 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:50.871 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 856 k 2023-10-21T03:58:50.872 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T03:58:50.872 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-21T03:58:50.872 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:50.872 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:50.872 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:50.872 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2023-10-21T03:58:50.873 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:50.873 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 934 k 2023-10-21T03:58:50.873 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 2.7 M 2023-10-21T03:58:50.874 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:50.977 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 752 kB/s | 79 kB 00:00 2023-10-21T03:58:51.160 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): ceph-fuse-18.0.0-6744.g7f103c7c.el9.x86_ 2.9 MB/s | 856 kB 00:00 2023-10-21T03:58:51.162 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:51.162 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.1 MB/s | 934 kB 00:00 2023-10-21T03:58:51.162 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:51.184 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:51.184 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:51.265 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:51.266 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:51.450 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-21T03:58:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 2/2 2023-10-21T03:58:52.313 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 2/2 2023-10-21T03:58:52.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/2 2023-10-21T03:58:52.621 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-21T03:58:52.622 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:52.622 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:52.622 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-21T03:58:52.622 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:52.622 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:52.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-volume 2023-10-21T03:58:53.449 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:53.595 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:53.596 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:53.597 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-21T03:58:53.597 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:53.597 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:53.597 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:18.0.0-6744.g7f103c7c.el9 ceph-noarch 262 k 2023-10-21T03:58:53.597 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:53.598 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:53.598 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:53.598 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:58:53.598 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:53.598 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 262 k 2023-10-21T03:58:53.599 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.2 M 2023-10-21T03:58:53.599 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:53.825 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-volume-18.0.0-6744.g7f103c7c.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-21T03:58:53.826 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:53.826 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-21T03:58:53.826 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:53.834 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:53.874 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:53.875 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:54.100 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:54.152 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:54.168 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:54.168 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:58:54.168 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-21T03:58:54.168 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:55.267 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:18.0.0-6744.g7f103c7c.el9.noarch 1/1 2023-10-21T03:58:55.267 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:55.268 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:55.268 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:18.0.0-6744.g7f103c7c.el9.noarch 2023-10-21T03:58:55.268 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:55.268 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:55.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2023-10-21T03:58:56.171 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:56.317 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:56.318 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:56.318 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-21T03:58:56.318 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:56.318 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:56.318 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 127 k 2023-10-21T03:58:56.319 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:56.319 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:56.319 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:56.319 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:58:56.319 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:56.319 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 127 k 2023-10-21T03:58:56.320 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 456 k 2023-10-21T03:58:56.320 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:56.530 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-18.0.0-6744.g7f103c7c.el9.x86_64 598 kB/s | 127 kB 00:00 2023-10-21T03:58:56.531 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:56.531 INFO:teuthology.orchestra.run.smithi040.stdout:Total 594 kB/s | 127 kB 00:00 2023-10-21T03:58:56.532 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:56.537 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:56.537 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:56.601 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:56.602 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:56.757 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:58:56.873 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:58:57.316 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:58:57.601 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:58:57.601 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:57.601 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:58:57.602 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:58:57.602 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:57.602 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:57.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2023-10-21T03:58:58.441 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:58.535 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:58:58.587 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:58.588 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:58:58.589 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:58:58.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2023-10-21T03:58:59.228 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:58:59.374 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:58:59.375 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-21T03:58:59.376 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:59.376 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:58:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 31 k 2023-10-21T03:58:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:59.377 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:58:59.377 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:58:59.377 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:58:59.377 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:58:59.378 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 31 k 2023-10-21T03:58:59.378 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 139 k 2023-10-21T03:58:59.378 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:58:59.776 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-18.0.0-6744.g7f103c7c.el9.x86_6 78 kB/s | 31 kB 00:00 2023-10-21T03:58:59.777 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:58:59.777 INFO:teuthology.orchestra.run.smithi040.stdout:Total 78 kB/s | 31 kB 00:00 2023-10-21T03:58:59.778 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:58:59.780 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:58:59.781 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:58:59.811 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:58:59.812 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:58:59.909 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:59:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:00.411 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:00.679 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:00.679 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:00.680 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:59:00.680 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:59:00.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:00.680 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:00.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2023-10-21T03:59:01.455 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:01.550 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:59:01.602 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:01.603 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:59:01.603 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:01.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2023-10-21T03:59:02.251 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:02.344 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:59:02.396 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:02.397 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:59:02.397 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:02.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados 2023-10-21T03:59:03.035 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:59:03.181 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:03.182 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:59:03.182 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:03.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw 2023-10-21T03:59:03.832 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:03.925 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:59:03.977 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:03.978 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:59:03.979 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:04.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs 2023-10-21T03:59:04.614 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:59:04.760 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:04.761 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:59:04.761 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:04.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd 2023-10-21T03:59:05.409 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:05.503 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T03:59:05.555 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:05.556 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-21T03:59:05.556 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:05.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2023-10-21T03:59:06.198 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:06.343 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:06.344 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:06.345 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-10-21T03:59:06.345 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:06.345 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:59:06.345 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 85 k 2023-10-21T03:59:06.345 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:06.346 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:59:06.346 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:06.346 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:59:06.346 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:06.346 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 85 k 2023-10-21T03:59:06.346 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 227 k 2023-10-21T03:59:06.347 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:59:06.654 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-18.0.0-6744.g7f103c7c.el9.x86_64.rpm 275 kB/s | 85 kB 00:00 2023-10-21T03:59:06.655 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:59:06.655 INFO:teuthology.orchestra.run.smithi040.stdout:Total 273 kB/s | 85 kB 00:00 2023-10-21T03:59:06.655 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:59:06.662 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:59:06.663 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:59:06.721 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:59:06.722 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:59:06.863 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:59:06.971 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:07.347 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:07.677 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:07.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:59:07.679 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:59:07.679 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:07.896 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2023-10-21T03:59:08.481 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:08.627 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:08.628 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:08.628 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-21T03:59:08.628 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:08.628 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:59:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 3.0 M 2023-10-21T03:59:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:08.629 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:59:08.629 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:08.629 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:59:08.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:08.630 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.0 M 2023-10-21T03:59:08.630 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 12 M 2023-10-21T03:59:08.630 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:59:08.998 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-18.0.0-6744.g7f103c7c.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2023-10-21T03:59:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:59:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:Total 8.1 MB/s | 3.0 MB 00:00 2023-10-21T03:59:09.000 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:59:09.009 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:59:09.009 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:59:09.085 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:59:09.086 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:59:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:59:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:09.608 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-21T03:59:09.608 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-21T03:59:09.608 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-21T03:59:09.608 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-21T03:59:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:10.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:10.694 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:59:10.694 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:59:10.694 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:10.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2023-10-21T03:59:11.528 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Oct 2023 03:57:27 AM UTC. 2023-10-21T03:59:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T03:59:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-10-21T03:59:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T03:59:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:18.0.0-6744.g7f103c7c.el9 ceph 169 k 2023-10-21T03:59:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T03:59:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T03:59:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-21T03:59:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 169 k 2023-10-21T03:59:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 486 k 2023-10-21T03:59:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T03:59:11.919 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-18.0.0-6744.g7f103c7c.el9.x86_64.rpm 702 kB/s | 169 kB 00:00 2023-10-21T03:59:11.920 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T03:59:11.920 INFO:teuthology.orchestra.run.smithi040.stdout:Total 698 kB/s | 169 kB 00:00 2023-10-21T03:59:11.920 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T03:59:11.928 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T03:59:11.928 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T03:59:11.987 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T03:59:11.988 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T03:59:12.146 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T03:59:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:13.007 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 1/1 2023-10-21T03:59:13.007 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T03:59:13.007 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 2023-10-21T03:59:13.008 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T03:59:13.178 DEBUG:teuthology.parallel:result is None 2023-10-21T03:59:13.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:59:13.403 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-21T03:59:13.425 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-6744.g7f103c7c.el9 2023-10-21T03:59:13.425 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744.g7f103c7c.el9 2023-10-21T03:59:13.426 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744.g7f103c7c is installed. 2023-10-21T03:59:13.426 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-21T03:59:13.427 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:13.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-21T03:59:13.496 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-21T03:59:13.496 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:13.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-10-21T03:59:13.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-21T03:59:13.637 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-21T03:59:13.637 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:13.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-21T03:59:13.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-21T03:59:13.777 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-21T03:59:13.778 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:13.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2023-10-21T03:59:13.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-21T03:59:13.943 INFO:teuthology.run_tasks:Running task ceph... 2023-10-21T03:59:14.019 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-21T03:59:14.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2023-10-21T03:59:14.051 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-21T03:59:14.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-21T03:59:14.123 INFO:tasks.ceph:Creating extra log directories... 2023-10-21T03:59:14.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-21T03:59:14.208 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-21T03:59:14.208 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-21T03:59:14.208 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357', 'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7433357', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ajarr-wip-extra-debug-62994-oct-19'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1449, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'ajarr-wip-extra-debug-62994-oct-19', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAfZOxwwOSOg7SAAW1HKu994xk9gt1F1rsH50yetsOqqRSjPNPyrelDEyxFF3hzsEyq0o6huOWqCT9bTgCmORpc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-20_15:15:00', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-21T03:59:14.209 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-21T03:59:14.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2023-10-21T03:59:14.322 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:14.322 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2023-10-21T03:59:14.378 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-21T03:59:14.378 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2023-10-21T03:59:14.434 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-21T03:59:14.434 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T03:59:14.434 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-10-21T03:59:14.434 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T03:59:14.435 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-21T03:59:14.435 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-21 03:59:09.841996744 +0000 2023-10-21T03:59:14.435 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-21 03:56:40.195909784 +0000 2023-10-21T03:59:14.435 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-21 03:56:40.195909784 +0000 2023-10-21T03:59:14.435 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-21 03:56:40.195909784 +0000 2023-10-21T03:59:14.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-21T03:59:14.503 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-21T03:59:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-21T03:59:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.00020421 s, 2.5 MB/s 2023-10-21T03:59:14.505 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-21T03:59:14.560 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2023-10-21T03:59:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-21T03:59:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T03:59:14.616 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-21T03:59:14.616 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T03:59:14.617 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-21T03:59:14.617 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-21 03:59:09.841996744 +0000 2023-10-21T03:59:14.617 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-21 03:56:40.712903360 +0000 2023-10-21T03:59:14.617 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-21 03:56:40.712903360 +0000 2023-10-21T03:59:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-21 03:56:40.712903360 +0000 2023-10-21T03:59:14.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-21T03:59:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-21T03:59:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-21T03:59:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000187357 s, 2.7 MB/s 2023-10-21T03:59:14.687 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-21T03:59:14.742 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2023-10-21T03:59:14.797 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-21T03:59:14.798 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T03:59:14.798 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-21T03:59:14.798 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T03:59:14.798 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-21T03:59:14.798 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-21 03:59:09.842996731 +0000 2023-10-21T03:59:14.799 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-21 03:56:41.259896563 +0000 2023-10-21T03:59:14.799 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-21 03:56:41.259896563 +0000 2023-10-21T03:59:14.799 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-21 03:56:41.259896563 +0000 2023-10-21T03:59:14.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-21T03:59:14.866 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-21T03:59:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-21T03:59:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000187749 s, 2.7 MB/s 2023-10-21T03:59:14.868 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-21T03:59:14.923 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2023-10-21T03:59:14.977 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-21T03:59:14.978 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T03:59:14.978 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-21T03:59:14.978 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T03:59:14.978 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-21T03:59:14.979 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-21 03:59:09.842996731 +0000 2023-10-21T03:59:14.979 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-21 03:56:41.782890064 +0000 2023-10-21T03:59:14.979 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-21 03:56:41.782890064 +0000 2023-10-21T03:59:14.979 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-21 03:56:41.782890064 +0000 2023-10-21T03:59:14.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-21T03:59:15.049 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-21T03:59:15.050 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-21T03:59:15.050 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000221518 s, 2.3 MB/s 2023-10-21T03:59:15.051 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-21T03:59:15.107 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T03:59:15.108 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T03:59:15.108 INFO:tasks.ceph:Generating config... 2023-10-21T03:59:15.109 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-21T03:59:15.109 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-21T03:59:15.110 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-21T03:59:15.110 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-21T03:59:15.110 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-21T03:59:15.111 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-21T03:59:15.111 INFO:tasks.ceph:[mgr] debug monc = 20 2023-10-21T03:59:15.111 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-21T03:59:15.111 INFO:tasks.ceph:[mgr] debug objecter = 20 2023-10-21T03:59:15.112 INFO:tasks.ceph:[mgr] debug rados = 20 2023-10-21T03:59:15.112 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-21T03:59:15.112 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-21T03:59:15.113 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-21T03:59:15.113 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-21T03:59:15.113 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-21T03:59:15.113 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-10-21T03:59:15.114 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-21T03:59:15.114 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-21T03:59:15.114 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-21T03:59:15.115 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-21T03:59:15.115 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-21T03:59:15.115 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-21T03:59:15.115 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-21T03:59:15.116 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-21T03:59:15.116 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-21T03:59:15.116 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-21T03:59:15.116 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-21T03:59:15.117 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-21T03:59:15.117 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-21T03:59:15.117 INFO:tasks.ceph:Setting up mon.a... 2023-10-21T03:59:15.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-21T03:59:15.201 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2023-10-21T03:59:15.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-21T03:59:15.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T03:59:15.281 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40')] 2023-10-21T03:59:15.282 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 20, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-21T03:59:15.283 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:15.283 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T03:59:15.337 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:15.415 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:15.416 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid df4211a3-b1f8-4292-becf-1d62cb6e7aab 2023-10-21T03:59:15.416 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = pacific 2023-10-21T03:59:15.416 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2023-10-21T03:59:15.416 INFO:teuthology.orchestra.run.smithi040.stdout:fsid df4211a3-b1f8-4292-becf-1d62cb6e7aab 2023-10-21T03:59:15.416 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2023-10-21T03:59:15.415071+0000 2023-10-21T03:59:15.417 INFO:teuthology.orchestra.run.smithi040.stdout:created 2023-10-21T03:59:15.415071+0000 2023-10-21T03:59:15.417 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 16 (pacific) 2023-10-21T03:59:15.417 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2023-10-21T03:59:15.417 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-10-21T03:59:15.417 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-21T03:59:15.419 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T03:59:15.474 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID df4211a3-b1f8-4292-becf-1d62cb6e7aab... 2023-10-21T03:59:15.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-21T03:59:15.574 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2023-10-21T03:59:15.574 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2023-10-21T03:59:15.575 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-21T03:59:15.575 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2023-10-21T03:59:15.575 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.575 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2023-10-21T03:59:15.575 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.576 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2023-10-21T03:59:15.576 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.576 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2023-10-21T03:59:15.576 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2023-10-21T03:59:15.576 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.577 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2023-10-21T03:59:15.577 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2023-10-21T03:59:15.577 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.577 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-21T03:59:15.577 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2023-10-21T03:59:15.578 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.578 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2023-10-21T03:59:15.578 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-21T03:59:15.578 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.578 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2023-10-21T03:59:15.579 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.579 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2023-10-21T03:59:15.579 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2023-10-21T03:59:15.579 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2023-10-21T03:59:15.579 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2023-10-21T03:59:15.579 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2023-10-21T03:59:15.580 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2023-10-21T03:59:15.580 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2023-10-21T03:59:15.580 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-21T03:59:15.580 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-21T03:59:15.580 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2023-10-21T03:59:15.581 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.581 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-21T03:59:15.581 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.581 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2023-10-21T03:59:15.581 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.581 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2023-10-21T03:59:15.582 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.582 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2023-10-21T03:59:15.582 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2023-10-21T03:59:15.582 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2023-10-21T03:59:15.582 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = 172.21.15.40 2023-10-21T03:59:15.583 INFO:teuthology.orchestra.run.smithi040.stdout: mon client directed command retry = 5 2023-10-21T03:59:15.583 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on pool no app = False 2023-10-21T03:59:15.583 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject socket failures = 5000 2023-10-21T03:59:15.583 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = df4211a3-b1f8-4292-becf-1d62cb6e7aab 2023-10-21T03:59:15.583 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.583 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2023-10-21T03:59:15.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2023-10-21T03:59:15.584 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2023-10-21T03:59:15.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2023-10-21T03:59:15.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2023-10-21T03:59:15.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.585 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2023-10-21T03:59:15.585 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2023-10-21T03:59:15.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.585 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2023-10-21T03:59:15.585 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2023-10-21T03:59:15.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2023-10-21T03:59:15.586 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2023-10-21T03:59:15.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2023-10-21T03:59:15.586 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2023-10-21T03:59:15.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2023-10-21T03:59:15.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.587 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2023-10-21T03:59:15.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.588 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2023-10-21T03:59:15.588 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2023-10-21T03:59:15.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.588 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2023-10-21T03:59:15.588 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2023-10-21T03:59:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2023-10-21T03:59:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression algorithm = zlib 2023-10-21T03:59:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression mode = aggressive 2023-10-21T03:59:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2023-10-21T03:59:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 1/20 2023-10-21T03:59:15.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 1/20 2023-10-21T03:59:15.590 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-21T03:59:15.590 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2023-10-21T03:59:15.590 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 4/10 2023-10-21T03:59:15.590 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2023-10-21T03:59:15.590 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2023-10-21T03:59:15.591 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2023-10-21T03:59:15.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2023-10-21T03:59:15.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2023-10-21T03:59:15.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2023-10-21T03:59:15.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.591 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2023-10-21T03:59:15.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-21T03:59:15.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2023-10-21T03:59:15.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-10-21T03:59:15.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-10-21T03:59:15.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-10-21T03:59:15.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-10-21T03:59:15.593 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2023-10-21T03:59:15.593 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2023-10-21T03:59:15.593 INFO:teuthology.orchestra.run.smithi040.stdout: debug objecter = 20 2023-10-21T03:59:15.593 INFO:teuthology.orchestra.run.smithi040.stdout: debug rados = 20 2023-10-21T03:59:15.593 INFO:teuthology.orchestra.run.smithi040.stdout: debug rbd = 20 2023-10-21T03:59:15.594 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.594 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2023-10-21T03:59:15.594 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-21T03:59:15.594 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-10-21T03:59:15.594 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2023-10-21T03:59:15.594 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-10-21T03:59:15.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2023-10-21T03:59:15.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2023-10-21T03:59:15.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-10-21T03:59:15.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2023-10-21T03:59:15.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2023-10-21T03:59:15.596 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-10-21T03:59:15.596 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.596 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-21T03:59:15.596 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2023-10-21T03:59:15.596 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2023-10-21T03:59:15.596 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.597 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2023-10-21T03:59:15.597 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-21T03:59:15.597 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-21T03:59:15.597 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.597 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2023-10-21T03:59:15.598 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2023-10-21T03:59:15.598 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.598 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2023-10-21T03:59:15.598 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:15.598 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2023-10-21T03:59:15.599 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2023-10-21T03:59:15.599 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2023-10-21T03:59:15.599 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2023-10-21T03:59:15.599 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-21T03:59:15.599 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-21T03:59:15.599 INFO:teuthology.orchestra.run.smithi040.stdout: rbd default data pool = datapool 2023-10-21T03:59:15.600 INFO:teuthology.orchestra.run.smithi040.stdout: rbd default features = 1 2023-10-21T03:59:15.600 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2023-10-21T03:59:15.601 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-21T03:59:15.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-21T03:59:15.700 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-21T03:59:15.700 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:15.701 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-21T03:59:15.716 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:15.716 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-21T03:59:15.770 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T03:59:15.771 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:15.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-21T03:59:15.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T03:59:15.857 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:15.858 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:15.913 INFO:tasks.ceph:Setting up mon nodes... 2023-10-21T03:59:15.913 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-21T03:59:15.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T03:59:16.014 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T03:59:16.016 INFO:tasks.ceph:Setting up mds nodes... 2023-10-21T03:59:16.017 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-21T03:59:16.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-21T03:59:16.063 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-21T03:59:16.082 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-21T03:59:16.082 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T03:59:16.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-21T03:59:16.113 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T03:59:16.113 INFO:tasks.ceph:role: osd.0 2023-10-21T03:59:16.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T03:59:16.114 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-21T03:59:16.191 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T03:59:16.191 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T03:59:16.192 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T03:59:16.192 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-21T03:59:16.192 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T03:59:16.192 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-21T03:59:16.193 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T03:59:16.193 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-21T03:59:16.193 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T03:59:16.193 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T03:59:16.693 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-21T03:59:16.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-21T03:59:16.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-21T03:59:16.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-21T03:59:16.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-21T03:59:16.944 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T03:59:16.945 INFO:tasks.ceph:role: osd.1 2023-10-21T03:59:16.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T03:59:16.945 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-21T03:59:17.024 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T03:59:17.025 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T03:59:17.025 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T03:59:17.025 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-21T03:59:17.025 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T03:59:17.026 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-21T03:59:17.026 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T03:59:17.026 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-21T03:59:17.026 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T03:59:17.027 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T03:59:17.478 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-21T03:59:17.478 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-21T03:59:17.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-21T03:59:17.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-21T03:59:17.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-21T03:59:17.691 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T03:59:17.692 INFO:tasks.ceph:role: osd.2 2023-10-21T03:59:17.692 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T03:59:17.692 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-21T03:59:17.772 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T03:59:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T03:59:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T03:59:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-21T03:59:17.773 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T03:59:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-21T03:59:17.773 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T03:59:17.774 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-21T03:59:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T03:59:17.774 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T03:59:18.258 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-21T03:59:18.260 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-21T03:59:18.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-21T03:59:18.397 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-21T03:59:18.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:18.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:18.536+0000 7f8086a1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-21T03:59:18.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:18.540+0000 7f8086a1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-21T03:59:18.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:18.540+0000 7f8086a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T03:59:18.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:18.540+0000 7f8086a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T03:59:18.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:18.540+0000 7f8086a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T03:59:18.550 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:18.548+0000 7f8086a1a500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-21T03:59:21.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-21T03:59:21.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:21.529+0000 7f7fe1210500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-21T03:59:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:21.530+0000 7f7fe1210500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-21T03:59:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:21.530+0000 7f7fe1210500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T03:59:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:21.530+0000 7f7fe1210500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T03:59:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:21.530+0000 7f7fe1210500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T03:59:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:21.530+0000 7f7fe1210500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-21T03:59:24.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-21T03:59:24.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:24.516+0000 7fd9a28f6500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-21T03:59:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:24.516+0000 7fd9a28f6500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-21T03:59:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:24.516+0000 7fd9a28f6500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T03:59:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:24.516+0000 7fd9a28f6500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T03:59:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:24.516+0000 7fd9a28f6500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T03:59:24.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T03:59:24.517+0000 7fd9a28f6500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-21T03:59:27.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-21T03:59:27.418 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-21T03:59:27.419 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:27.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-21T03:59:27.492 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:27.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-21T03:59:27.570 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:27.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-21T03:59:27.643 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:27.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-21T03:59:27.716 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:27.717 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-21T03:59:27.773 INFO:tasks.ceph:Adding keys to all mons... 2023-10-21T03:59:27.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-21T03:59:27.843 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2023-10-21T03:59:27.843 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCUTDNlfMfTABAA1oAv917GKOQ4j5noe0cccw== 2023-10-21T03:59:27.844 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2023-10-21T03:59:27.844 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCWTDNlHtsRIBAA+/zO9CLuYwN4j9u8cBAPKQ== 2023-10-21T03:59:27.844 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2023-10-21T03:59:27.845 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCZTDNlCyiqHxAAj1WGvYeOFA54HUXhGL4lwA== 2023-10-21T03:59:27.845 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2023-10-21T03:59:27.845 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCcTDNlwMzbHhAAspKcRfkENjSplIAtVQoeKw== 2023-10-21T03:59:27.846 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2023-10-21T03:59:27.846 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCUTDNlb8rBAxAAmTXSGJw0mhzlK9mbrjIlvg== 2023-10-21T03:59:27.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-21T03:59:27.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T03:59:28.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T03:59:28.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T03:59:28.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-21T03:59:28.173 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-21T03:59:28.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-21T03:59:28.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-21T03:59:28.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-21T03:59:28.338 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-21T03:59:28.393 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-21T03:59:28.393 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-21T03:59:28.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-21T03:59:28.435 INFO:tasks.ceph.mon.a:Started 2023-10-21T03:59:28.436 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-21T03:59:28.436 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-21T03:59:28.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-21T03:59:28.439 INFO:tasks.ceph.mgr.x:Started 2023-10-21T03:59:28.440 INFO:tasks.ceph:Setting crush tunables to default 2023-10-21T03:59:28.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2023-10-21T03:59:28.633 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2023-10-21T03:59:28.644 INFO:tasks.ceph:check_enable_crimson: False 2023-10-21T03:59:28.644 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-21T03:59:28.644 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:28.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-21T03:59:28.675 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:28.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-21T03:59:28.746 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:28.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-21T03:59:28.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new c5f8202a-b1a1-454a-b079-217103b4c22b 0 2023-10-21T03:59:29.048 INFO:teuthology.orchestra.run.smithi040.stdout:0 2023-10-21T03:59:29.058 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new b3ec9f98-f470-4211-af84-4d4565e48ddc 1 2023-10-21T03:59:29.227 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:29.225+0000 7f4ea2e10040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-21T03:59:29.266 INFO:teuthology.orchestra.run.smithi040.stdout:1 2023-10-21T03:59:29.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new b7c30dc1-691d-4c84-a137-16ad328be016 2 2023-10-21T03:59:29.282 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:29.280+0000 7f4ea2e10040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-21T03:59:29.512 INFO:teuthology.orchestra.run.smithi040.stdout:2 2023-10-21T03:59:29.523 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-21T03:59:29.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-21T03:59:29.565 INFO:tasks.ceph.osd.0:Started 2023-10-21T03:59:29.566 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-21T03:59:29.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-21T03:59:29.569 INFO:tasks.ceph.osd.1:Started 2023-10-21T03:59:29.569 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-21T03:59:29.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-21T03:59:29.573 INFO:tasks.ceph.osd.2:Started 2023-10-21T03:59:29.575 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T03:59:29.767 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:29.767 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T03:59:29.510751+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T03:59:29.778 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-21T03:59:29.778 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-21T03:59:29.894 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:29.892+0000 7f4ea2e10040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-21T03:59:29.949 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:29.947+0000 7f4ea2e10040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-21T03:59:29.995 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:29.993+0000 7f4ea2e10040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-21T03:59:30.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.057+0000 7f4ea2e10040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-21T03:59:30.083 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T03:59:30.104 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.101+0000 7f4ea2e10040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-21T03:59:30.167 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T03:59:30.165+0000 7f1e50ab6500 -1 Falling back to public interface 2023-10-21T03:59:30.193 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T03:59:30.191+0000 7f562aa14500 -1 Falling back to public interface 2023-10-21T03:59:30.194 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T03:59:30.191+0000 7f00ec40d500 -1 Falling back to public interface 2023-10-21T03:59:30.194 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T03:59:30.192+0000 7f56292f8640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2023-10-21T03:59:30.238 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.236+0000 7f4ea2e10040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-21T03:59:30.276 INFO:teuthology.misc.health.smithi040.stdout: 2023-10-21T03:59:30.276 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":5,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T03:59:29.510751+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T03:59:30.286 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T03:59:30.314 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.312+0000 7f4ea2e10040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-21T03:59:30.362 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.360+0000 7f4ea2e10040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-21T03:59:30.416 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.414+0000 7f4ea2e10040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-21T03:59:30.465 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.463+0000 7f4ea2e10040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-21T03:59:30.861 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:30.859+0000 7f4ea2e10040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-21T03:59:31.003 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.001+0000 7f4ea2e10040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-21T03:59:31.120 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.118+0000 7f4ea2e10040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-21T03:59:31.167 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.165+0000 7f4ea2e10040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-21T03:59:31.270 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.266+0000 7f4ea2e10040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-21T03:59:31.368 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.365+0000 7f4ea2e10040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-21T03:59:31.511 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.509+0000 7f4ea2e10040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-21T03:59:31.556 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.553+0000 7f4ea2e10040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-21T03:59:31.701 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:31.699+0000 7f4ea2e10040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-21T03:59:32.532 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:32.530+0000 7f4ea2e10040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-21T03:59:32.816 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:32.814+0000 7f4ea2e10040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-21T03:59:32.869 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:32.867+0000 7f4ea2e10040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-21T03:59:32.994 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:32.992+0000 7f4ea2e10040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-21T03:59:33.647 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T03:59:33.645+0000 7f1e50ab6500 -1 osd.0 0 log_to_monitors true 2023-10-21T03:59:33.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T03:59:33.680+0000 7f00ec40d500 -1 osd.1 0 log_to_monitors true 2023-10-21T03:59:33.709 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T03:59:33.707+0000 7f562aa14500 -1 osd.2 0 log_to_monitors true 2023-10-21T03:59:34.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T03:59:34.012+0000 7f4e43e69640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-21T03:59:36.588 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T03:59:36.888 INFO:teuthology.misc.health.smithi040.stdout: 2023-10-21T03:59:36.888 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":7,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T03:59:35.526791+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T03:59:36.900 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T03:59:38.825 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T03:59:38.823+0000 7f56262dd640 -1 osd.2 0 waiting for initial osdmap 2023-10-21T03:59:38.830 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T03:59:38.828+0000 7f00e7dc9640 -1 osd.1 0 waiting for initial osdmap 2023-10-21T03:59:38.836 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T03:59:38.834+0000 7f561e0fe640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T03:59:38.843 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T03:59:38.841+0000 7f00dfbea640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T03:59:38.851 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T03:59:38.850+0000 7f1e4ca42640 -1 osd.0 0 waiting for initial osdmap 2023-10-21T03:59:38.870 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T03:59:38.868+0000 7f1e44863640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T03:59:43.203 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T03:59:43.493 INFO:teuthology.misc.health.smithi040.stdout: 2023-10-21T03:59:43.493 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T03:59:42.543716+0000","last_up_change":"2023-10-21T03:59:39.830650+0000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T03:59:41.070351+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6801","nonce":3061035859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6803","nonce":3061035859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6807","nonce":3061035859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6805","nonce":3061035859}]},"public_addr":"172.21.15.40:6801/3061035859","cluster_addr":"172.21.15.40:6803/3061035859","heartbeat_back_addr":"172.21.15.40:6807/3061035859","heartbeat_front_addr":"172.21.15.40:6805/3061035859","state":["exists","up"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6810","nonce":1845782567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6814","nonce":1845782567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6821","nonce":1845782567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6818","nonce":1845782567}]},"public_addr":"172.21.15.40:6810/1845782567","cluster_addr":"172.21.15.40:6814/1845782567","heartbeat_back_addr":"172.21.15.40:6821/1845782567","heartbeat_front_addr":"172.21.15.40:6818/1845782567","state":["exists","up"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6811","nonce":3642351925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6815","nonce":3642351925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6823","nonce":3642351925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6820","nonce":3642351925}]},"public_addr":"172.21.15.40:6811/3642351925","cluster_addr":"172.21.15.40:6815/3642351925","heartbeat_back_addr":"172.21.15.40:6823/3642351925","heartbeat_front_addr":"172.21.15.40:6820/3642351925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.602779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.659903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.753221+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T03:59:43.504 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-21T03:59:43.505 INFO:tasks.ceph:Creating RBD pool 2023-10-21T03:59:43.505 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-21T03:59:44.526 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'rbd' created 2023-10-21T03:59:44.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-21T03:59:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-21T03:59:45.547 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-21T03:59:45.548 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-21T03:59:45.548 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-21T03:59:45.549 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-21T03:59:45.549 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-21T03:59:45.901 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:45.912 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2395648111},{"type":"v1","addr":"172.21.15.40:6825","nonce":2395648111}]},"active_addr":"172.21.15.40:6825/2395648111","active_change":"2023-10-21T03:59:33.006226+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":4090375840}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":1779402539}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3779173012}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3760626093}]}]} 2023-10-21T03:59:45.916 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-21T03:59:45.916 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-21T03:59:45.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T03:59:46.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:46.208 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T03:59:45.525739+0000","last_up_change":"2023-10-21T03:59:39.830650+0000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T03:59:41.070351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T03:59:43.852830+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6801","nonce":3061035859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6803","nonce":3061035859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6807","nonce":3061035859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6805","nonce":3061035859}]},"public_addr":"172.21.15.40:6801/3061035859","cluster_addr":"172.21.15.40:6803/3061035859","heartbeat_back_addr":"172.21.15.40:6807/3061035859","heartbeat_front_addr":"172.21.15.40:6805/3061035859","state":["exists","up"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6810","nonce":1845782567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6814","nonce":1845782567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6821","nonce":1845782567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6818","nonce":1845782567}]},"public_addr":"172.21.15.40:6810/1845782567","cluster_addr":"172.21.15.40:6814/1845782567","heartbeat_back_addr":"172.21.15.40:6821/1845782567","heartbeat_front_addr":"172.21.15.40:6818/1845782567","state":["exists","up"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6811","nonce":3642351925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6815","nonce":3642351925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6823","nonce":3642351925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6820","nonce":3642351925}]},"public_addr":"172.21.15.40:6811/3642351925","cluster_addr":"172.21.15.40:6815/3642351925","heartbeat_back_addr":"172.21.15.40:6823/3642351925","heartbeat_front_addr":"172.21.15.40:6820/3642351925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.602779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.659903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.753221+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T03:59:46.220 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-21T03:59:46.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T03:59:46.516 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:46.517 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T03:59:45.525739+0000","last_up_change":"2023-10-21T03:59:39.830650+0000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T03:59:41.070351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T03:59:43.852830+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6801","nonce":3061035859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6803","nonce":3061035859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6807","nonce":3061035859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6805","nonce":3061035859}]},"public_addr":"172.21.15.40:6801/3061035859","cluster_addr":"172.21.15.40:6803/3061035859","heartbeat_back_addr":"172.21.15.40:6807/3061035859","heartbeat_front_addr":"172.21.15.40:6805/3061035859","state":["exists","up"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6810","nonce":1845782567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6814","nonce":1845782567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6821","nonce":1845782567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6818","nonce":1845782567}]},"public_addr":"172.21.15.40:6810/1845782567","cluster_addr":"172.21.15.40:6814/1845782567","heartbeat_back_addr":"172.21.15.40:6821/1845782567","heartbeat_front_addr":"172.21.15.40:6818/1845782567","state":["exists","up"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6811","nonce":3642351925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6815","nonce":3642351925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6823","nonce":3642351925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6820","nonce":3642351925}]},"public_addr":"172.21.15.40:6811/3642351925","cluster_addr":"172.21.15.40:6815/3642351925","heartbeat_back_addr":"172.21.15.40:6823/3642351925","heartbeat_front_addr":"172.21.15.40:6820/3642351925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.602779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.659903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.753221+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T03:59:46.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-21T03:59:46.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-21T03:59:46.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-21T03:59:46.676 INFO:teuthology.orchestra.run.smithi040.stdout:34359738374 2023-10-21T03:59:46.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T03:59:46.678 INFO:teuthology.orchestra.run.smithi040.stdout:34359738375 2023-10-21T03:59:46.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T03:59:46.679 INFO:teuthology.orchestra.run.smithi040.stdout:34359738375 2023-10-21T03:59:46.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T03:59:47.032 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-21T03:59:47.033 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-21T03:59:47.035 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-21T03:59:47.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738373 for osd.0 2023-10-21T03:59:47.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738373 for osd.1 2023-10-21T03:59:47.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.2 2023-10-21T03:59:48.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T03:59:48.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T03:59:48.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T03:59:48.365 INFO:teuthology.orchestra.run.smithi040.stdout:34359738375 2023-10-21T03:59:48.372 INFO:teuthology.orchestra.run.smithi040.stdout:34359738375 2023-10-21T03:59:48.377 INFO:teuthology.orchestra.run.smithi040.stdout:34359738374 2023-10-21T03:59:48.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2023-10-21T03:59:48.378 DEBUG:teuthology.parallel:result is None 2023-10-21T03:59:48.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2023-10-21T03:59:48.386 DEBUG:teuthology.parallel:result is None 2023-10-21T03:59:48.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2023-10-21T03:59:48.390 DEBUG:teuthology.parallel:result is None 2023-10-21T03:59:48.390 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T03:59:48.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T03:59:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T03:59:48.730 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-10-21T03:59:47.012590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81496,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283034024,"statfs":{"total":289910292480,"available":289826840576,"internally_reserved":0,"allocated":434176,"data_stored":1266332,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.485073"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528756+0000","last_change":"2023-10-21T03:59:45.537738+0000","last_active":"2023-10-21T03:59:46.528756+0000","last_peered":"2023-10-21T03:59:46.528756+0000","last_clean":"2023-10-21T03:59:46.528756+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T03:59:46.528756+0000","last_undegraded":"2023-10-21T03:59:46.528756+0000","last_fullsized":"2023-10-21T03:59:46.528756+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:27:35.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528687+0000","last_change":"2023-10-21T03:59:45.537129+0000","last_active":"2023-10-21T03:59:46.528687+0000","last_peered":"2023-10-21T03:59:46.528687+0000","last_clean":"2023-10-21T03:59:46.528687+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T03:59:46.528687+0000","last_undegraded":"2023-10-21T03:59:46.528687+0000","last_fullsized":"2023-10-21T03:59:46.528687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:05:01.589410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528654+0000","last_change":"2023-10-21T03:59:45.537287+0000","last_active":"2023-10-21T03:59:46.528654+0000","last_peered":"2023-10-21T03:59:46.528654+0000","last_clean":"2023-10-21T03:59:46.528654+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T03:59:46.528654+0000","last_undegraded":"2023-10-21T03:59:46.528654+0000","last_fullsized":"2023-10-21T03:59:46.528654+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:28:54.781889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528626+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T03:59:46.528626+0000","last_peered":"2023-10-21T03:59:46.528626+0000","last_clean":"2023-10-21T03:59:46.528626+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T03:59:46.528626+0000","last_undegraded":"2023-10-21T03:59:46.528626+0000","last_fullsized":"2023-10-21T03:59:46.528626+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:10:11.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.534966+0000","last_change":"2023-10-21T03:59:45.534408+0000","last_active":"2023-10-21T03:59:45.534966+0000","last_peered":"2023-10-21T03:59:45.534966+0000","last_clean":"2023-10-21T03:59:45.534966+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T03:59:45.534966+0000","last_undegraded":"2023-10-21T03:59:45.534966+0000","last_fullsized":"2023-10-21T03:59:45.534966+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:36:47.399191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.538971+0000","last_change":"2023-10-21T03:59:45.534361+0000","last_active":"2023-10-21T03:59:45.538971+0000","last_peered":"2023-10-21T03:59:45.538971+0000","last_clean":"2023-10-21T03:59:45.538971+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T03:59:45.538971+0000","last_undegraded":"2023-10-21T03:59:45.538971+0000","last_fullsized":"2023-10-21T03:59:45.538971+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:27:28.172697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.538951+0000","last_change":"2023-10-21T03:59:45.534393+0000","last_active":"2023-10-21T03:59:45.538951+0000","last_peered":"2023-10-21T03:59:45.538951+0000","last_clean":"2023-10-21T03:59:45.538951+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T03:59:45.538951+0000","last_undegraded":"2023-10-21T03:59:45.538951+0000","last_fullsized":"2023-10-21T03:59:45.538951+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:28:30.997461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528622+0000","last_change":"2023-10-21T03:59:45.536989+0000","last_active":"2023-10-21T03:59:46.528622+0000","last_peered":"2023-10-21T03:59:46.528622+0000","last_clean":"2023-10-21T03:59:46.528622+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T03:59:46.528622+0000","last_undegraded":"2023-10-21T03:59:46.528622+0000","last_fullsized":"2023-10-21T03:59:46.528622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:54:11.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.539074+0000","last_change":"2023-10-21T03:59:43.546458+0000","last_active":"2023-10-21T03:59:45.539074+0000","last_peered":"2023-10-21T03:59:45.539074+0000","last_clean":"2023-10-21T03:59:45.539074+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T03:59:45.539074+0000","last_undegraded":"2023-10-21T03:59:45.539074+0000","last_fullsized":"2023-10-21T03:59:45.539074+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_clean_scrub_stamp":"2023-10-21T03:59:41.541284+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:25:24.228129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":106496,"data_stored":28532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":618900,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.16800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":618900,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T03:59:48.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T03:59:49.022 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T03:59:49.033 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-21T03:59:49.012879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81496,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283034024,"statfs":{"total":289910292480,"available":289826840576,"internally_reserved":0,"allocated":434176,"data_stored":1266332,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.485362"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528756+0000","last_change":"2023-10-21T03:59:45.537738+0000","last_active":"2023-10-21T03:59:46.528756+0000","last_peered":"2023-10-21T03:59:46.528756+0000","last_clean":"2023-10-21T03:59:46.528756+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T03:59:46.528756+0000","last_undegraded":"2023-10-21T03:59:46.528756+0000","last_fullsized":"2023-10-21T03:59:46.528756+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:27:35.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528687+0000","last_change":"2023-10-21T03:59:45.537129+0000","last_active":"2023-10-21T03:59:46.528687+0000","last_peered":"2023-10-21T03:59:46.528687+0000","last_clean":"2023-10-21T03:59:46.528687+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T03:59:46.528687+0000","last_undegraded":"2023-10-21T03:59:46.528687+0000","last_fullsized":"2023-10-21T03:59:46.528687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:05:01.589410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528654+0000","last_change":"2023-10-21T03:59:45.537287+0000","last_active":"2023-10-21T03:59:46.528654+0000","last_peered":"2023-10-21T03:59:46.528654+0000","last_clean":"2023-10-21T03:59:46.528654+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T03:59:46.528654+0000","last_undegraded":"2023-10-21T03:59:46.528654+0000","last_fullsized":"2023-10-21T03:59:46.528654+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:28:54.781889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528626+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T03:59:46.528626+0000","last_peered":"2023-10-21T03:59:46.528626+0000","last_clean":"2023-10-21T03:59:46.528626+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T03:59:46.528626+0000","last_undegraded":"2023-10-21T03:59:46.528626+0000","last_fullsized":"2023-10-21T03:59:46.528626+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:10:11.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.534966+0000","last_change":"2023-10-21T03:59:45.534408+0000","last_active":"2023-10-21T03:59:45.534966+0000","last_peered":"2023-10-21T03:59:45.534966+0000","last_clean":"2023-10-21T03:59:45.534966+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T03:59:45.534966+0000","last_undegraded":"2023-10-21T03:59:45.534966+0000","last_fullsized":"2023-10-21T03:59:45.534966+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:36:47.399191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.538971+0000","last_change":"2023-10-21T03:59:45.534361+0000","last_active":"2023-10-21T03:59:45.538971+0000","last_peered":"2023-10-21T03:59:45.538971+0000","last_clean":"2023-10-21T03:59:45.538971+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T03:59:45.538971+0000","last_undegraded":"2023-10-21T03:59:45.538971+0000","last_fullsized":"2023-10-21T03:59:45.538971+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:27:28.172697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.538951+0000","last_change":"2023-10-21T03:59:45.534393+0000","last_active":"2023-10-21T03:59:45.538951+0000","last_peered":"2023-10-21T03:59:45.538951+0000","last_clean":"2023-10-21T03:59:45.538951+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T03:59:45.538951+0000","last_undegraded":"2023-10-21T03:59:45.538951+0000","last_fullsized":"2023-10-21T03:59:45.538951+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:28:30.997461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T03:59:46.528622+0000","last_change":"2023-10-21T03:59:45.536989+0000","last_active":"2023-10-21T03:59:46.528622+0000","last_peered":"2023-10-21T03:59:46.528622+0000","last_clean":"2023-10-21T03:59:46.528622+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T03:59:46.528622+0000","last_undegraded":"2023-10-21T03:59:46.528622+0000","last_fullsized":"2023-10-21T03:59:46.528622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:54:11.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T03:59:45.539074+0000","last_change":"2023-10-21T03:59:43.546458+0000","last_active":"2023-10-21T03:59:45.539074+0000","last_peered":"2023-10-21T03:59:45.539074+0000","last_clean":"2023-10-21T03:59:45.539074+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T03:59:45.539074+0000","last_undegraded":"2023-10-21T03:59:45.539074+0000","last_fullsized":"2023-10-21T03:59:45.539074+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_clean_scrub_stamp":"2023-10-21T03:59:41.541284+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:25:24.228129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":106496,"data_stored":28532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":618900,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.16800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":618900,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T03:59:49.035 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T03:59:49.035 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-21T03:59:49.035 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-21T03:59:49.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-21T03:59:49.364 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T03:59:49.364 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-21T03:59:49.376 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-21T03:59:49.376 INFO:teuthology.run_tasks:Running task exec... 2023-10-21T03:59:49.385 INFO:teuthology.task.exec:Executing custom commands... 2023-10-21T03:59:49.386 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T03:59:49.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-21T03:59:50.574 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'datapool' created 2023-10-21T03:59:50.593 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-21T03:59:53.614 INFO:teuthology.run_tasks:Running task install... 2023-10-21T03:59:53.623 DEBUG:teuthology.task.install:project ceph 2023-10-21T03:59:53.623 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T03:59:53.624 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T03:59:53.624 INFO:teuthology.task.install:Using flavor: default 2023-10-21T03:59:53.631 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T03:59:53.632 INFO:teuthology.task.install:extra packages: [] 2023-10-21T03:59:53.633 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['fio'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'tag': None, 'wait_for_package': False} 2023-10-21T03:59:53.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:59:53.779 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/ 2023-10-21T03:59:53.780 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6744.g7f103c7c 2023-10-21T03:59:53.962 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-21T03:59:53.962 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T03:59:53.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-21T03:59:53.999 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, fio, fio on remote rpm x86_64 2023-10-21T03:59:54.000 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-21T03:59:54.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-21T03:59:54.148 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-21T03:59:54.205 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2023-10-21T03:59:54.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2023-10-21T03:59:54.474 INFO:teuthology.orchestra.run.smithi040.stdout:73 files removed 2023-10-21T03:59:54.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd fio fio 2023-10-21T03:59:55.320 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 165 kB/s | 78 kB 00:00 2023-10-21T03:59:55.696 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 29 kB/s | 9.4 kB 00:00 2023-10-21T03:59:56.055 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2023-10-21T03:59:56.457 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 17 MB/s | 6.3 MB 00:00 2023-10-21T03:59:57.850 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2023-10-21T03:59:59.552 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 7.9 MB 00:01 2023-10-21T04:00:01.334 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 26 MB/s | 18 MB 00:00 2023-10-21T04:00:07.788 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 5.7 MB 00:03 2023-10-21T04:00:09.435 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2023-10-21T04:00:12.004 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 19 MB 00:02 2023-10-21T04:00:16.696 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-10-21T04:00:19.148 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-radosgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.149 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-test-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.150 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.151 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.152 INFO:teuthology.orchestra.run.smithi040.stdout:Package cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch is already installed. 2023-10-21T04:00:19.153 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-immutable-object-cache-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.154 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.155 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-dashboard-2:18.0.0-6744.g7f103c7c.el9.noarch is already installed. 2023-10-21T04:00:19.157 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-diskprediction-local-2:18.0.0-6744.g7f103c7c.el9.noarch is already installed. 2023-10-21T04:00:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-rook-2:18.0.0-6744.g7f103c7c.el9.noarch is already installed. 2023-10-21T04:00:19.159 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-cephadm-2:18.0.0-6744.g7f103c7c.el9.noarch is already installed. 2023-10-21T04:00:19.160 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.160 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-volume-2:18.0.0-6744.g7f103c7c.el9.noarch is already installed. 2023-10-21T04:00:19.161 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.163 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.163 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs-devel-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.164 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.165 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.166 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.167 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.169 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.170 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.171 INFO:teuthology.orchestra.run.smithi040.stdout:Package rbd-fuse-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.172 INFO:teuthology.orchestra.run.smithi040.stdout:Package rbd-mirror-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.173 INFO:teuthology.orchestra.run.smithi040.stdout:Package rbd-nbd-2:18.0.0-6744.g7f103c7c.el9.x86_64 is already installed. 2023-10-21T04:00:19.228 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-21T04:00:19.229 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T04:00:19.229 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-10-21T04:00:19.230 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T04:00:19.230 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-21T04:00:19.230 INFO:teuthology.orchestra.run.smithi040.stdout: fio x86_64 3.35-1.el9 appstream 6.2 M 2023-10-21T04:00:19.230 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-21T04:00:19.230 INFO:teuthology.orchestra.run.smithi040.stdout: libnbd x86_64 1.16.0-1.el9 appstream 161 k 2023-10-21T04:00:19.231 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2023-10-21T04:00:19.231 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-http x86_64 3.35-1.el9 appstream 18 k 2023-10-21T04:00:19.231 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-libaio x86_64 3.35-1.el9 appstream 16 k 2023-10-21T04:00:19.231 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-nbd x86_64 3.35-1.el9 appstream 15 k 2023-10-21T04:00:19.231 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-rados x86_64 3.35-1.el9 appstream 16 k 2023-10-21T04:00:19.231 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-rbd x86_64 3.35-1.el9 appstream 16 k 2023-10-21T04:00:19.232 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-rdma x86_64 3.35-1.el9 appstream 21 k 2023-10-21T04:00:19.232 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T04:00:19.232 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-21T04:00:19.232 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-21T04:00:19.232 INFO:teuthology.orchestra.run.smithi040.stdout:Install 8 Packages 2023-10-21T04:00:19.233 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T04:00:19.233 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 6.5 M 2023-10-21T04:00:19.234 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 9.1 M 2023-10-21T04:00:19.234 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-21T04:00:19.788 INFO:teuthology.orchestra.run.smithi040.stdout:(1/8): fio-engine-libaio-3.35-1.el9.x86_64.rpm 54 kB/s | 16 kB 00:00 2023-10-21T04:00:19.814 INFO:teuthology.orchestra.run.smithi040.stdout:(2/8): fio-engine-http-3.35-1.el9.x86_64.rpm 58 kB/s | 18 kB 00:00 2023-10-21T04:00:19.955 INFO:teuthology.orchestra.run.smithi040.stdout:(3/8): fio-engine-nbd-3.35-1.el9.x86_64.rpm 87 kB/s | 15 kB 00:00 2023-10-21T04:00:19.989 INFO:teuthology.orchestra.run.smithi040.stdout:(4/8): fio-engine-rados-3.35-1.el9.x86_64.rpm 105 kB/s | 16 kB 00:00 2023-10-21T04:00:20.106 INFO:teuthology.orchestra.run.smithi040.stdout:(5/8): fio-engine-rbd-3.35-1.el9.x86_64.rpm 108 kB/s | 16 kB 00:00 2023-10-21T04:00:20.140 INFO:teuthology.orchestra.run.smithi040.stdout:(6/8): fio-engine-rdma-3.35-1.el9.x86_64.rpm 165 kB/s | 21 kB 00:00 2023-10-21T04:00:20.340 INFO:teuthology.orchestra.run.smithi040.stdout:(7/8): fio-3.35-1.el9.x86_64.rpm 7.4 MB/s | 6.2 MB 00:00 2023-10-21T04:00:20.382 INFO:teuthology.orchestra.run.smithi040.stdout:(8/8): libnbd-1.16.0-1.el9.x86_64.rpm 587 kB/s | 161 kB 00:00 2023-10-21T04:00:20.382 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-21T04:00:20.383 INFO:teuthology.orchestra.run.smithi040.stdout:Total 5.6 MB/s | 6.5 MB 00:01 2023-10-21T04:00:20.512 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-21T04:00:20.525 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-21T04:00:20.525 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-21T04:00:20.652 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-21T04:00:20.653 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-21T04:00:21.041 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-21T04:00:21.101 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libnbd-1.16.0-1.el9.x86_64 1/8 2023-10-21T04:00:21.126 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-engine-http-3.35-1.el9.x86_64 2/8 2023-10-21T04:00:21.160 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-engine-libaio-3.35-1.el9.x86_64 3/8 2023-10-21T04:00:21.185 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-engine-nbd-3.35-1.el9.x86_64 4/8 2023-10-21T04:00:21.210 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-engine-rados-3.35-1.el9.x86_64 5/8 2023-10-21T04:00:21.236 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-engine-rbd-3.35-1.el9.x86_64 6/8 2023-10-21T04:00:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-engine-rdma-3.35-1.el9.x86_64 7/8 2023-10-21T04:00:21.496 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fio-3.35-1.el9.x86_64 8/8 2023-10-21T04:00:22.737 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: fio-3.35-1.el9.x86_64 8/8 2023-10-21T04:00:22.738 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-3.35-1.el9.x86_64 1/8 2023-10-21T04:00:22.738 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-engine-http-3.35-1.el9.x86_64 2/8 2023-10-21T04:00:22.738 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-engine-libaio-3.35-1.el9.x86_64 3/8 2023-10-21T04:00:22.738 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-engine-nbd-3.35-1.el9.x86_64 4/8 2023-10-21T04:00:22.738 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-engine-rados-3.35-1.el9.x86_64 5/8 2023-10-21T04:00:22.739 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-engine-rbd-3.35-1.el9.x86_64 6/8 2023-10-21T04:00:22.739 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fio-engine-rdma-3.35-1.el9.x86_64 7/8 2023-10-21T04:00:23.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libnbd-1.16.0-1.el9.x86_64 8/8 2023-10-21T04:00:23.118 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T04:00:23.118 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-21T04:00:23.118 INFO:teuthology.orchestra.run.smithi040.stdout: fio-3.35-1.el9.x86_64 fio-engine-http-3.35-1.el9.x86_64 2023-10-21T04:00:23.118 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-libaio-3.35-1.el9.x86_64 fio-engine-nbd-3.35-1.el9.x86_64 2023-10-21T04:00:23.119 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-rados-3.35-1.el9.x86_64 fio-engine-rbd-3.35-1.el9.x86_64 2023-10-21T04:00:23.119 INFO:teuthology.orchestra.run.smithi040.stdout: fio-engine-rdma-3.35-1.el9.x86_64 libnbd-1.16.0-1.el9.x86_64 2023-10-21T04:00:23.119 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T04:00:23.119 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-21T04:00:23.477 DEBUG:teuthology.parallel:result is None 2023-10-21T04:00:23.478 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:00:23.736 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-21T04:00:23.759 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-6744.g7f103c7c.el9 2023-10-21T04:00:23.760 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744.g7f103c7c.el9 2023-10-21T04:00:23.760 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744.g7f103c7c is installed. 2023-10-21T04:00:23.760 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-21T04:00:23.761 INFO:teuthology.run_tasks:Running task workunit... 2023-10-21T04:00:23.775 INFO:tasks.workunit:Pulling workunits from ref 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:00:23.775 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-21T04:00:23.776 DEBUG:teuthology.orchestra.run.smithi040:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:00:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:00:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-21T04:00:23.817 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:00:23.872 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-21T04:00:23.873 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-21T04:00:23.928 INFO:tasks.workunit:timeout=3h 2023-10-21T04:00:23.928 INFO:tasks.workunit:cleanup=True 2023-10-21T04:00:23.929 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:00:23.985 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-21T04:01:34.376 INFO:tasks.workunit.client.0.smithi040.stderr:Updating files: 99% (11981/12067) Updating files: 100% (12067/12067) Updating files: 100% (12067/12067), done. 2023-10-21T04:01:34.809 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'. 2023-10-21T04:01:34.809 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.809 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-21T04:01:34.809 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-21T04:01:34.810 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2023-10-21T04:01:34.810 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.810 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-21T04:01:34.810 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-21T04:01:34.810 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.811 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2023-10-21T04:01:34.811 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.811 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2023-10-21T04:01:34.811 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.811 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2023-10-21T04:01:34.812 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.812 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-21T04:01:34.812 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-10-21T04:01:34.812 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at 7f103c7cad2 qa/suites/rbd/cli/workloads/rbd_support_module_recovery: more debug 2023-10-21T04:01:34.814 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-21T04:01:34.875 INFO:tasks.workunit.client.0.smithi040.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-21T04:01:34.877 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:01:34.877 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-21T04:01:35.543 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-21T04:01:35.608 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-21T04:01:35.648 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:01:35.650 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:01:35.650 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-21T04:01:35.690 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:01:35.693 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-21T04:01:35.693 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-21T04:01:35.749 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-21T04:01:35.750 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-21T04:01:35.750 DEBUG:teuthology.orchestra.run.smithi040:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-21T04:01:35.807 INFO:tasks.workunit.client.0.smithi040.stderr:+ POOL=rbd 2023-10-21T04:01:35.808 INFO:tasks.workunit.client.0.smithi040.stderr:+ IMAGE_PREFIX=image 2023-10-21T04:01:35.808 INFO:tasks.workunit.client.0.smithi040.stderr:+ NUM_IMAGES=20 2023-10-21T04:01:35.808 INFO:tasks.workunit.client.0.smithi040.stderr:+ RUN_TIME=3600 2023-10-21T04:01:35.808 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror pool enable rbd image 2023-10-21T04:01:35.845 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-21T04:01:35.881 INFO:tasks.workunit.client.0.smithi040.stdout:759b707c-2f7b-4843-8800-f6db66e6bff8 2023-10-21T04:01:35.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i = 1 )) 2023-10-21T04:01:35.883 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:35.883 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-21T04:01:35.929 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-21T04:01:36.701 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:36.707 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-21T04:01:36.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:36.762 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:36.762 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-21T04:01:36.810 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-21T04:01:37.704 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:37.711 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-21T04:01:37.765 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:37.765 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:37.766 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-21T04:01:37.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-21T04:01:38.708 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:38.716 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-21T04:01:38.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:38.771 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:38.771 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-21T04:01:38.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-21T04:01:39.713 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:39.722 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-21T04:01:39.775 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:39.775 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:39.776 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-21T04:01:39.819 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-21T04:01:40.719 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:40.726 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-21T04:01:40.779 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:40.780 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:40.780 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-21T04:01:40.825 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-21T04:01:41.727 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:41.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-21T04:01:41.785 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:41.785 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:41.786 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-21T04:01:41.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-21T04:01:42.732 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:42.740 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-21T04:01:42.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:42.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:42.794 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-21T04:01:42.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-21T04:01:43.737 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:43.744 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-21T04:01:43.798 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:43.798 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:43.799 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-21T04:01:43.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-21T04:01:44.737 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:44.744 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-21T04:01:44.798 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:44.798 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:44.799 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-21T04:01:44.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-21T04:01:45.742 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:45.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-21T04:01:45.803 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:45.803 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:45.803 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-21T04:01:45.851 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-21T04:01:46.754 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:46.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-21T04:01:46.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:46.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:46.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-21T04:01:46.861 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-21T04:01:47.751 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:47.757 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-21T04:01:47.811 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:47.812 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:47.812 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-21T04:01:47.860 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-21T04:01:48.753 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:48.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-21T04:01:48.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:48.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:48.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-21T04:01:48.864 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-21T04:01:49.760 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:49.766 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-21T04:01:49.821 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:49.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:49.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-21T04:01:49.866 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-21T04:01:50.764 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:50.772 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-21T04:01:50.828 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:50.828 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:50.828 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-21T04:01:50.873 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-21T04:01:51.773 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:51.780 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-21T04:01:51.833 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:51.834 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:51.834 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-21T04:01:51.878 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-21T04:01:52.781 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:52.791 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-21T04:01:52.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:52.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:52.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-21T04:01:52.888 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-21T04:01:53.778 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:53.785 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-21T04:01:53.839 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:53.839 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:53.839 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-21T04:01:53.883 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-21T04:01:54.778 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:54.785 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-21T04:01:54.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:54.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:54.839 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-21T04:01:54.884 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-21T04:01:55.781 INFO:tasks.workunit.client.0.smithi040.stdout:Mirroring enabled 2023-10-21T04:01:55.788 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-21T04:01:55.841 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:55.842 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:55.842 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i = 1 )) 2023-10-21T04:01:55.842 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:55.842 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image1 2023-10-21T04:01:56.003 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-21T04:01:56.003 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.004 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image2 2023-10-21T04:01:56.071 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-21T04:01:56.072 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.072 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.072 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.072 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image3 2023-10-21T04:01:56.140 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-21T04:01:56.141 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.141 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.141 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.141 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image4 2023-10-21T04:01:56.222 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-21T04:01:56.223 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.223 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.223 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.223 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image5 2023-10-21T04:01:56.304 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-21T04:01:56.304 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.304 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.304 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.305 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image6 2023-10-21T04:01:56.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-21T04:01:56.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.422 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image7 2023-10-21T04:01:56.544 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-21T04:01:56.544 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.544 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.544 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.545 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image8 2023-10-21T04:01:56.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-21T04:01:56.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.669 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.669 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image9 2023-10-21T04:01:56.814 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-21T04:01:56.815 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.815 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.815 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:56.816 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image10 2023-10-21T04:01:56.968 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-21T04:01:56.969 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:56.969 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:56.970 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image11 2023-10-21T04:01:56.970 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:57.122 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-21T04:01:57.122 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:57.122 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:57.123 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image12 2023-10-21T04:01:57.123 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:57.260 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-21T04:01:57.260 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:57.261 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:57.261 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:57.261 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image13 2023-10-21T04:01:57.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-21T04:01:57.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:57.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:57.413 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:57.413 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image14 2023-10-21T04:01:57.578 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-21T04:01:57.579 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:57.579 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:57.579 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:57.579 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image15 2023-10-21T04:01:57.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-21T04:01:57.735 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:57.735 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:57.735 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:57.735 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image16 2023-10-21T04:01:57.881 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-21T04:01:57.881 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:57.881 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:57.881 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image17 2023-10-21T04:01:57.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:58.034 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-21T04:01:58.034 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:58.035 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:58.036 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:58.036 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image18 2023-10-21T04:01:58.383 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-21T04:01:58.384 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:58.384 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:58.385 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:58.385 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image19 2023-10-21T04:01:58.565 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-21T04:01:58.566 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:58.566 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:58.567 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:58.567 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo rbd device map rbd/image20 2023-10-21T04:01:58.727 INFO:tasks.workunit.client.0.smithi040.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-21T04:01:58.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T04:01:58.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 20 )) 2023-10-21T04:01:58.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:01:58.728 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:01:58.729 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860918 2023-10-21T04:01:58.729 INFO:tasks.workunit.client.0.smithi040.stderr:+ END_TIME=1697864518 2023-10-21T04:01:58.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR= 2023-10-21T04:01:58.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR= 2023-10-21T04:01:58.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:01:58.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n '' ]] 2023-10-21T04:01:58.731 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:02:00.557 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:00.554+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:02:00.558 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:00.554+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:02:00.558 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:00.554+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:02:00.559 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:00.554+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:02:05.560 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:05.558+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:02:05.561 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:05.558+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:02:08.731 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:02:08.731 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:02:08.732 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:02:08.732 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:02:09.669 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3779173012 2023-10-21T04:02:09.669 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:02:09.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860929 2023-10-21T04:02:09.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:02:09.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3779173012 ]] 2023-10-21T04:02:09.671 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3779173012 != '' ]] 2023-10-21T04:02:09.671 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3779173012 2023-10-21T04:02:09.792 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:09.789+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:02:09.792 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:09.789+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:02:09.793 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:09.789+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:02:09.793 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:09.789+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:02:11.583 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3779173012 until 2023-10-21T05:02:10.799486+0000 (3600 sec) 2023-10-21T04:02:11.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:02:11.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3779173012 2023-10-21T04:02:12.350 INFO:tasks.workunit.client.0.smithi040.stderr:listed 1 entries 2023-10-21T04:02:12.370 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3779173012 2023-10-21T04:02:12.370 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:02:22.377 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:02:22.377 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:02:22.377 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:02:22.377 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:02:23.271 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3779173012 2023-10-21T04:02:23.272 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:02:23.273 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860943 2023-10-21T04:02:23.273 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:02:23.273 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3779173012 ]] 2023-10-21T04:02:23.273 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3779173012 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\7\9\1\7\3\0\1\2 ]] 2023-10-21T04:02:23.273 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:02:33.277 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:02:33.278 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:02:33.278 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:02:33.279 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:02:34.205 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3779173012 2023-10-21T04:02:34.205 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:02:34.207 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860954 2023-10-21T04:02:34.207 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:02:34.207 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3779173012 ]] 2023-10-21T04:02:34.207 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3779173012 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\7\9\1\7\3\0\1\2 ]] 2023-10-21T04:02:34.207 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:02:44.209 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:02:44.209 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:02:44.209 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:02:44.216 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:02:45.120 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:02:45.120 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:02:45.121 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860965 2023-10-21T04:02:45.121 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:02:45.121 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:02:45.121 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\7\9\1\7\3\0\1\2 ]] 2023-10-21T04:02:45.122 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2556549987 2023-10-21T04:02:46.674 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2556549987 until 2023-10-21T05:02:45.890209+0000 (3600 sec) 2023-10-21T04:02:46.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:02:46.696 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2556549987 2023-10-21T04:02:47.449 INFO:tasks.workunit.client.0.smithi040.stderr:listed 2 entries 2023-10-21T04:02:47.477 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:02:47.477 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:02:48.972 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:02:48.972 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:02:48.972 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:02:48.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:02:48.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:02:48.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:02:48.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:02:48.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:02:48.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:02:48.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:02:48.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:02:48.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:02:48.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:02:48.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:02:48.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:02:48.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:02:48.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:02:48.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:02:48.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:02:53.647 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:02:53.645+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:02:57.479 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:02:57.479 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:02:57.479 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:02:57.481 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:02:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.112+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:02:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.112+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:02:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.112+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:02:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.112+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:02:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.112+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:02:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.112+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:02:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.113+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:02:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.113+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:02:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.113+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:02:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:02:58.113+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:02:58.409 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:02:58.410 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:02:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860978 2023-10-21T04:02:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:02:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:02:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:02:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:03:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.006+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.006+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f27a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f97f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.016+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.016+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.021+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.022+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.027+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.027+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f94d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.031+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.031+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.036+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.036+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f0c5780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.042+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.042+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.045+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.045+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.051+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.051+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f27a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.052+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.052+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f94d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.057+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.057+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.060+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.061+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.064+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.064+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.067+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.067+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f27ac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.074+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.074+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288ef9be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.074+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.074+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f0c5880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.078+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.078+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.082+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.082+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.090 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.089+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:00.090 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:03:00.089+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:03:06.172 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:06.170+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:03:06.173 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:06.170+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:03:06.173 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:06.170+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:03:08.413 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:03:08.413 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:08.415 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:08.415 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:09.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:03:09.335 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:03:09.336 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697860989 2023-10-21T04:03:09.336 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:09.336 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:03:09.337 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:03:09.337 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:03:12.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:12.677+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:03:12.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:12.677+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:03:19.338 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:03:19.339 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:19.339 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:19.339 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:20.260 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:03:20.261 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:03:20.261 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861000 2023-10-21T04:03:20.261 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:20.261 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:03:20.262 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:03:20.262 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:03:20.899 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:03:20.899 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:03:20.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:03:20.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:03:20.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:03:20.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:03:20.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:03:20.901 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:03:20.901 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:03:20.901 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:20.896+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:03:22.292 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:03:22.292 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:03:22.293 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:03:22.293 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:03:22.293 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:03:22.293 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:03:22.293 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:03:22.289+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:03:22.391 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:22.388+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:03:22.391 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:22.388+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:03:22.391 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:22.388+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:03:30.267 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:03:30.267 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:30.267 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:30.268 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:31.208 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:03:31.208 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:03:31.209 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861011 2023-10-21T04:03:31.209 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:31.209 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:03:31.210 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:03:31.210 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:03:41.211 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:03:41.214 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:41.215 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:41.215 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:42.133 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:03:42.134 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:03:42.134 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861022 2023-10-21T04:03:42.134 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:42.135 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:03:42.135 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:03:42.135 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:03:44.701 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:03:44.699+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:03:52.136 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:03:52.139 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:52.140 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:52.140 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:53.075 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2556549987 2023-10-21T04:03:53.076 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:03:53.076 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861033 2023-10-21T04:03:53.077 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:53.077 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2556549987 ]] 2023-10-21T04:03:53.077 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2556549987 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:03:53.077 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:04:03.079 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:04:03.080 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:03.080 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:03.080 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:04.024 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:04:04.024 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:04:04.025 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861044 2023-10-21T04:04:04.026 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:04.026 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1185537480 ]] 2023-10-21T04:04:04.026 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1185537480 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\5\6\5\4\9\9\8\7 ]] 2023-10-21T04:04:04.026 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1185537480 2023-10-21T04:04:05.648 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1185537480 until 2023-10-21T05:04:04.860258+0000 (3600 sec) 2023-10-21T04:04:05.664 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:04:05.664 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1185537480 2023-10-21T04:04:06.041 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:06.032+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:04:06.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:06.032+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:04:06.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:06.032+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:04:06.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:06.032+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:04:06.043 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:06.032+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:04:06.400 INFO:tasks.workunit.client.0.smithi040.stderr:listed 3 entries 2023-10-21T04:04:06.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:04:06.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:04:07.978 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.974+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:04:07.978 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.974+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:04:07.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.974+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:04:07.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.975+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:04:07.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.975+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:04:07.980 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.975+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:04:07.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.982+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:04:07.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.982+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:04:07.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.982+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:04:07.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:07.982+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:04:16.426 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:04:16.426 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:16.428 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:16.437 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:17.338 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:04:17.338 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:04:17.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861057 2023-10-21T04:04:17.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:17.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1185537480 ]] 2023-10-21T04:04:17.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1185537480 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\8\5\5\3\7\4\8\0 ]] 2023-10-21T04:04:17.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:04:27.340 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:04:27.340 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:27.343 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:27.343 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:28.256 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:04:28.257 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:04:28.258 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861068 2023-10-21T04:04:28.258 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:28.258 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1185537480 ]] 2023-10-21T04:04:28.258 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1185537480 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\8\5\5\3\7\4\8\0 ]] 2023-10-21T04:04:28.258 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:04:38.214 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:04:38.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:04:38.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:04:38.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:04:38.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:04:38.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:04:38.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:04:38.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:04:38.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:04:38.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:38.211+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:04:38.261 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:04:38.262 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:38.262 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:38.262 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:39.207 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:04:39.207 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:04:39.208 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861079 2023-10-21T04:04:39.208 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:39.208 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1185537480 ]] 2023-10-21T04:04:39.208 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1185537480 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\8\5\5\3\7\4\8\0 ]] 2023-10-21T04:04:39.208 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:04:49.210 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:04:49.210 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:49.210 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:49.212 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:49.311 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.308+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:04:49.311 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.308+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:04:49.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.310+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:04:49.313 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.310+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:04:49.313 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.310+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:04:49.313 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.310+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:04:49.313 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:04:49.310+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:04:50.147 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:04:50.148 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:04:50.149 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861090 2023-10-21T04:04:50.149 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:50.149 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1185537480 ]] 2023-10-21T04:04:50.149 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1185537480 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\8\5\5\3\7\4\8\0 ]] 2023-10-21T04:04:50.150 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:04:56.152 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:56.148+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:04:56.152 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:56.148+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:04:56.152 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:56.148+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:04:57.750 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:57.747+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:04:57.750 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:57.747+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:04:57.751 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:57.747+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:04:57.751 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:04:57.747+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:05:00.154 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:05:00.154 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:00.154 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:00.155 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:01.068 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1185537480 2023-10-21T04:05:01.068 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:05:01.069 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861101 2023-10-21T04:05:01.069 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:01.069 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1185537480 ]] 2023-10-21T04:05:01.069 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1185537480 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\8\5\5\3\7\4\8\0 ]] 2023-10-21T04:05:01.070 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:05:08.509 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:08.503+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:05:08.510 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:08.503+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:05:08.511 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:08.503+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:05:08.511 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:08.503+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:05:09.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:05:09.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:05:09.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:05:09.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:05:09.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:05:09.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:05:09.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:05:09.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:05:09.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:05:09.880 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:05:09.872+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:05:11.074 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:11.074 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:11.074 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:11.074 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:05:12.000 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:05:12.000 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:05:12.003 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861112 2023-10-21T04:05:12.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:12.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1168953340 ]] 2023-10-21T04:05:12.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1168953340 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\8\5\5\3\7\4\8\0 ]] 2023-10-21T04:05:12.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1168953340 2023-10-21T04:05:13.763 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1168953340 until 2023-10-21T05:05:12.985685+0000 (3600 sec) 2023-10-21T04:05:13.781 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:05:13.782 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1168953340 2023-10-21T04:05:14.542 INFO:tasks.workunit.client.0.smithi040.stderr:listed 4 entries 2023-10-21T04:05:14.566 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:05:14.566 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:05:24.571 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:05:24.571 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:24.572 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:24.581 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:25.506 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:05:25.507 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:05:25.507 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861125 2023-10-21T04:05:25.508 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:25.508 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1168953340 ]] 2023-10-21T04:05:25.508 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1168953340 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\6\8\9\5\3\3\4\0 ]] 2023-10-21T04:05:25.508 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:05:31.483 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:31.481+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:05:35.513 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:05:35.513 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:35.513 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:35.514 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:36.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:05:36.422 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:05:36.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861136 2023-10-21T04:05:36.422 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:36.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1168953340 ]] 2023-10-21T04:05:36.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1168953340 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\6\8\9\5\3\3\4\0 ]] 2023-10-21T04:05:36.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:05:46.424 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:05:46.425 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:46.425 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:46.425 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:47.344 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:05:47.345 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:05:47.345 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861147 2023-10-21T04:05:47.346 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:47.346 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1168953340 ]] 2023-10-21T04:05:47.346 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1168953340 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\6\8\9\5\3\3\4\0 ]] 2023-10-21T04:05:47.346 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:05:53.600 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:53.597+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:05:53.600 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:53.597+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:05:53.601 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:53.597+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:05:53.601 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:05:53.597+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:05:57.347 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:05:57.349 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:57.349 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:57.353 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:58.247 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:05:58.247 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:05:58.248 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861158 2023-10-21T04:05:58.248 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:58.248 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1168953340 ]] 2023-10-21T04:05:58.249 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1168953340 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\6\8\9\5\3\3\4\0 ]] 2023-10-21T04:05:58.249 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:06:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.005+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.005+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd58a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.008+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.008+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.015+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.015+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.022+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.022+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.022+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.022+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fd59c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.027+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.027+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd58d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.031+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.031+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890072580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.042+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.042+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890072380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.044+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.044+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.051+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.051+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd58000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.055+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.055+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd58780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.060+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.060+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.066+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.066+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.073+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.073+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.079+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.079+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.079+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.079+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890072400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.082+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.082+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890072980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.085+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.088 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.085+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890072b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.090 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.088+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.091 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.088+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890073400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.094 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.092+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.094 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:06:00.092+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890073280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:01.503 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:06:01.500+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:06:01.504 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:06:01.500+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:06:01.504 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:06:01.500+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:06:08.251 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:06:08.255 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:08.256 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:08.261 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:09.159 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1168953340 2023-10-21T04:06:09.159 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:06:09.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861169 2023-10-21T04:06:09.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:09.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1168953340 ]] 2023-10-21T04:06:09.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1168953340 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\6\8\9\5\3\3\4\0 ]] 2023-10-21T04:06:09.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:06:19.161 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:06:19.161 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:19.161 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:19.164 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:19.477 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:06:19.477 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:06:19.478 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:06:19.478 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:06:19.479 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:06:19.479 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:06:19.479 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:06:19.480 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:06:19.480 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:06:19.480 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:06:19.470+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:06:20.108 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:06:20.108 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:06:20.109 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861180 2023-10-21T04:06:20.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:20.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3701969523 ]] 2023-10-21T04:06:20.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3701969523 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\1\6\8\9\5\3\3\4\0 ]] 2023-10-21T04:06:20.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3701969523 2023-10-21T04:06:21.709 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3701969523 until 2023-10-21T05:06:20.928002+0000 (3600 sec) 2023-10-21T04:06:21.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:06:21.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3701969523 2023-10-21T04:06:22.472 INFO:tasks.workunit.client.0.smithi040.stderr:listed 5 entries 2023-10-21T04:06:22.492 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:06:22.492 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:06:32.494 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:06:32.497 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:32.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:32.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:33.437 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:06:33.438 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:06:33.438 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861193 2023-10-21T04:06:33.438 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:33.439 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3701969523 ]] 2023-10-21T04:06:33.439 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3701969523 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\1\9\6\9\5\2\3 ]] 2023-10-21T04:06:33.439 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:06:43.441 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:06:43.441 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:43.441 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:43.442 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:44.371 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:06:44.372 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:06:44.372 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861204 2023-10-21T04:06:44.373 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:44.373 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3701969523 ]] 2023-10-21T04:06:44.373 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3701969523 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\1\9\6\9\5\2\3 ]] 2023-10-21T04:06:44.373 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:06:54.374 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:06:54.375 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:54.375 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:54.375 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:55.312 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:06:55.312 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:06:55.313 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861215 2023-10-21T04:06:55.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:55.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3701969523 ]] 2023-10-21T04:06:55.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3701969523 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\1\9\6\9\5\2\3 ]] 2023-10-21T04:06:55.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:07:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.004+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.004+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fab8100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.008+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.008+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fab9200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.014+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.014+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f97fc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.017+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.017+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f97f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.020+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.020+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f97ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.024+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.024+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fd59380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.027+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.027+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f97e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.031+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.031+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fd59b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.036+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.036+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fd58f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.044+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.044+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f94de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.046+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.046+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f94da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.049+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.049+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f94d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.054+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.054+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890072d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.063+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.063+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890072f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.064+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.064+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.065+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.065+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.071+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.072+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.075+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.075+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.077+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.077+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.081+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:07:00.081+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.102 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:07:02.102 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:07:02.103 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:07:02.103 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:07:02.103 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:07:02.103 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:07:02.103 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:07:02.104 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:07:02.104 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:02.100+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:07:05.317 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:07:05.318 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:05.318 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:05.318 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:05.890 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:05.887+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:07:05.891 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:05.887+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:07:05.891 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:05.887+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:07:06.263 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:07:06.264 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:07:06.264 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861226 2023-10-21T04:07:06.264 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:06.264 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3701969523 ]] 2023-10-21T04:07:06.264 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3701969523 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\1\9\6\9\5\2\3 ]] 2023-10-21T04:07:06.265 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:07:13.148 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.147+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:07:13.148 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.147+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:07:13.149 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.147+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:07:13.149 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.147+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:07:13.824 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.821+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:07:13.824 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.821+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:07:13.825 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.821+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:07:13.825 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:13.821+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:07:16.265 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:16.266 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:07:16.267 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:16.268 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:17.199 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3701969523 2023-10-21T04:07:17.200 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:07:17.201 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861237 2023-10-21T04:07:17.201 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:17.202 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3701969523 ]] 2023-10-21T04:07:17.202 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3701969523 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\1\9\6\9\5\2\3 ]] 2023-10-21T04:07:17.202 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:07:20.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:20.126+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:07:27.206 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:07:27.206 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:27.206 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:27.207 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:28.141 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:07:28.142 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:07:28.143 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861248 2023-10-21T04:07:28.143 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:28.143 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/796892313 ]] 2023-10-21T04:07:28.143 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/796892313 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\1\9\6\9\5\2\3 ]] 2023-10-21T04:07:28.144 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/796892313 2023-10-21T04:07:30.827 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/796892313 until 2023-10-21T05:07:29.851866+0000 (3600 sec) 2023-10-21T04:07:30.849 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:07:30.849 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/796892313 2023-10-21T04:07:31.600 INFO:tasks.workunit.client.0.smithi040.stderr:listed 6 entries 2023-10-21T04:07:31.624 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:07:31.625 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:07:33.844 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:33.840+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:07:33.845 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:33.841+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:07:36.140 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:07:36.137+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:07:37.679 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.676+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:07:37.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:07:37.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:07:37.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:07:37.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:07:37.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:07:37.682 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:07:37.682 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:07:37.682 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:07:37.682 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:37.678+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:07:41.626 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:07:41.627 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:41.627 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:41.627 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:42.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:07:42.558 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:07:42.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861262 2023-10-21T04:07:42.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:42.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/796892313 ]] 2023-10-21T04:07:42.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/796892313 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\9\6\8\9\2\3\1\3 ]] 2023-10-21T04:07:42.560 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:07:44.721 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:44.719+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:07:44.722 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:44.719+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:07:44.723 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:44.719+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:07:52.561 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:07:52.561 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:52.563 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:52.563 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:53.187 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:07:53.182+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:07:53.508 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:07:53.509 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:07:53.510 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861273 2023-10-21T04:07:53.510 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:53.510 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/796892313 ]] 2023-10-21T04:07:53.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/796892313 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\9\6\8\9\2\3\1\3 ]] 2023-10-21T04:07:53.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:08:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.006+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.006+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.007+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.007+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.012+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.017+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.017+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.021+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.021+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.027+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.027+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.037+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.037+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe83f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.044+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.044+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.048+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.048+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.053+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.053+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.057+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.057+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.062+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.062+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.068+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.068+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.073+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.073+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.079+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.079+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.082+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.082+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.088 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.086+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.088 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.086+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.093 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.090+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.093 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.090+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.094+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:08:00.094+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.514 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:08:03.514 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:03.514 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:03.515 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:04.446 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:08:04.447 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:08:04.448 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861284 2023-10-21T04:08:04.448 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:04.448 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/796892313 ]] 2023-10-21T04:08:04.448 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/796892313 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\9\6\8\9\2\3\1\3 ]] 2023-10-21T04:08:04.448 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:08:14.452 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:08:14.452 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:14.453 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:14.456 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:15.391 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:08:15.392 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:08:15.393 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861295 2023-10-21T04:08:15.393 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:15.393 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/796892313 ]] 2023-10-21T04:08:15.394 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/796892313 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\9\6\8\9\2\3\1\3 ]] 2023-10-21T04:08:15.394 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:08:18.885 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:08:18.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:08:18.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:08:18.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:08:18.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:08:18.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:08:18.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:08:18.888 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:08:18.888 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:08:18.888 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:18.884+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:08:24.970 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:24.966+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:08:24.971 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:24.966+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:08:24.971 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:24.966+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:08:24.971 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:24.966+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:08:25.398 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:08:25.398 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:25.398 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:25.399 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:26.319 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/796892313 2023-10-21T04:08:26.321 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:08:26.322 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861306 2023-10-21T04:08:26.322 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:26.322 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/796892313 ]] 2023-10-21T04:08:26.323 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/796892313 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\9\6\8\9\2\3\1\3 ]] 2023-10-21T04:08:26.323 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:08:34.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:08:34.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:08:34.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:08:34.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:08:34.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:08:34.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:08:34.191 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:34.186+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:08:36.324 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:08:36.325 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:36.329 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:36.329 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:37.031 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:37.027+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:08:37.031 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:37.027+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:08:37.032 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:37.027+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:08:37.032 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:37.027+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:08:37.283 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:08:37.283 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:08:37.284 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861317 2023-10-21T04:08:37.284 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:37.284 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:08:37.285 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\9\6\8\9\2\3\1\3 ]] 2023-10-21T04:08:37.285 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2008281116 2023-10-21T04:08:39.743 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2008281116 until 2023-10-21T05:08:38.958992+0000 (3600 sec) 2023-10-21T04:08:39.765 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2008281116 2023-10-21T04:08:39.765 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:08:40.455 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:08:40.456 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:08:40.456 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:08:40.456 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:08:40.456 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:08:40.457 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:08:40.457 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:08:40.457 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:08:40.457 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:08:40.457 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:08:40.450+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:08:40.518 INFO:tasks.workunit.client.0.smithi040.stderr:listed 7 entries 2023-10-21T04:08:40.543 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:08:40.543 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:08:50.542 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:08:50.543 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:50.543 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:50.543 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:51.472 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:08:51.472 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:08:51.473 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861331 2023-10-21T04:08:51.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:51.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:08:51.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:08:51.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:08:53.707 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:53.705+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:08:53.708 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:53.705+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:08:53.708 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:53.705+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:08:53.708 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:08:53.705+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:09:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.005+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.005+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890073900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.011+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.011+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890072a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.013+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.013+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f768d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.015+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.016+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890072b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.020+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.020+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f769300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.027+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.027+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fd59e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.029+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.029+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f94da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.035+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.035+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fd58400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.039+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.039+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fd59280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.043+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.043+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f0c4f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.050+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.050+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f94c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.053+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.053+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289028a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.057+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.058+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289028a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.061+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.061+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289028a580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.066+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.066+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289028a780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.072+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.072+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.076+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.076+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028b280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.079+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.079+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.083+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.083+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.088 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.086+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.089 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:09:00.086+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028ad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:01.475 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:09:01.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:01.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:01.479 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:02.416 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:09:02.417 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:09:02.418 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861342 2023-10-21T04:09:02.418 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:02.418 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:09:02.418 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:09:02.419 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:09:12.420 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:09:12.420 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:12.421 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:12.435 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:13.355 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:09:13.355 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:09:13.356 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861353 2023-10-21T04:09:13.356 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:13.357 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:09:13.357 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:09:13.357 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:09:23.358 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:09:23.359 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:23.359 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:23.359 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:23.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:09:23.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:09:23.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:09:23.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:09:23.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:09:23.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:09:23.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:09:23.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:09:23.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:09:23.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:23.683+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:09:24.252 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:09:24.253 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:09:24.253 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861364 2023-10-21T04:09:24.254 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:24.254 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:09:24.254 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:09:24.254 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:09:34.255 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:09:34.257 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:34.258 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:34.258 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:35.178 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:09:35.178 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:09:35.179 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861375 2023-10-21T04:09:35.179 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:35.180 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:09:35.180 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:09:35.180 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:09:38.667 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.662+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:09:38.667 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.663+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:09:38.667 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.663+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:09:38.668 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.663+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:09:38.668 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.664+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:09:38.668 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.664+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:09:38.668 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.664+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:09:38.668 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.664+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:09:38.669 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:38.664+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:09:45.181 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:09:45.182 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:45.185 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:45.185 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:46.099 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2008281116 2023-10-21T04:09:46.099 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:09:46.100 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861386 2023-10-21T04:09:46.100 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:46.100 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2008281116 ]] 2023-10-21T04:09:46.101 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2008281116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:09:46.101 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:09:48.006 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:09:48.003+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:09:48.006 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:09:48.003+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:09:48.006 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:09:48.003+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:09:48.007 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:09:48.003+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:09:56.102 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:09:56.103 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:56.103 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:56.103 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:57.014 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:09:57.014 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:09:57.015 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861397 2023-10-21T04:09:57.015 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:57.015 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1373339116 ]] 2023-10-21T04:09:57.015 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1373339116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\0\0\8\2\8\1\1\1\6 ]] 2023-10-21T04:09:57.016 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1373339116 2023-10-21T04:09:57.746 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:57.743+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:09:57.746 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:09:57.743+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:09:58.684 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1373339116 until 2023-10-21T05:09:57.909039+0000 (3600 sec) 2023-10-21T04:09:58.704 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:09:58.705 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1373339116 2023-10-21T04:09:59.375 INFO:tasks.workunit.client.0.smithi040.stderr:listed 8 entries 2023-10-21T04:09:59.391 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:09:59.391 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:10:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.015+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.015+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f97ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.016+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.016+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.016+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe82f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.016+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe82e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.020+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.020+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe82b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.024+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.024+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f94dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.030+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.030+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890073e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.032+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.032+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fb8ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fab9380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.039+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.039+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fb8b600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.042+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.042+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fb8ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.052+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.052+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.052+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fb8af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.052+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890072980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f0c5580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fb8be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fb8ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.065+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.065+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f27ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.070+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.070+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f768880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.075+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.075+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f0c4f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.077+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.077+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.080+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:10:00.080+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289028ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:09.393 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:10:09.393 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:09.393 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:09.394 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:10.340 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:10:10.340 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:10:10.341 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861410 2023-10-21T04:10:10.341 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:10.342 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1373339116 ]] 2023-10-21T04:10:10.342 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1373339116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\3\7\3\3\3\9\1\1\6 ]] 2023-10-21T04:10:10.342 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:10:20.343 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:10:20.343 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:20.344 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:20.354 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:21.297 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:10:21.298 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:10:21.299 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861421 2023-10-21T04:10:21.299 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:21.299 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1373339116 ]] 2023-10-21T04:10:21.300 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1373339116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\3\7\3\3\3\9\1\1\6 ]] 2023-10-21T04:10:21.300 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:10:26.827 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:26.823+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:10:26.827 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:26.823+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:10:26.828 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:26.823+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:10:30.847 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:30.844+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:10:30.847 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:30.844+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:10:30.848 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:30.844+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:10:30.848 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:10:30.844+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:10:31.300 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:10:31.300 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:31.300 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:31.301 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:32.212 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:10:32.213 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:10:32.214 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861432 2023-10-21T04:10:32.214 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:32.214 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1373339116 ]] 2023-10-21T04:10:32.214 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1373339116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\3\7\3\3\3\9\1\1\6 ]] 2023-10-21T04:10:32.214 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:10:38.272 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.268+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:10:38.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:10:38.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:10:38.278 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:10:38.278 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:10:38.278 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:10:38.278 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:10:38.278 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:10:38.278 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:10:38.273+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:10:42.219 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:10:42.219 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:42.219 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:42.220 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:43.183 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:10:43.184 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:10:43.184 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861443 2023-10-21T04:10:43.184 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:43.185 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1373339116 ]] 2023-10-21T04:10:43.185 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1373339116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\3\7\3\3\3\9\1\1\6 ]] 2023-10-21T04:10:43.185 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:10:53.187 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:10:53.187 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:53.187 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:53.187 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:54.151 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1373339116 2023-10-21T04:10:54.152 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:10:54.153 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861454 2023-10-21T04:10:54.153 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:54.153 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1373339116 ]] 2023-10-21T04:10:54.153 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1373339116 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\3\7\3\3\3\9\1\1\6 ]] 2023-10-21T04:10:54.154 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:11:04.051 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:04.049+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:11:04.052 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:04.049+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:11:04.052 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:04.049+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:11:04.052 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:04.049+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:11:04.154 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:11:04.155 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:04.157 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:04.157 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:05.057 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:11:05.057 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:11:05.058 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861465 2023-10-21T04:11:05.059 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:05.059 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848573237 ]] 2023-10-21T04:11:05.059 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848573237 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\3\7\3\3\3\9\1\1\6 ]] 2023-10-21T04:11:05.060 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2848573237 2023-10-21T04:11:06.810 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2848573237 until 2023-10-21T05:11:06.041078+0000 (3600 sec) 2023-10-21T04:11:06.834 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:11:06.834 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2848573237 2023-10-21T04:11:07.599 INFO:tasks.workunit.client.0.smithi040.stderr:listed 9 entries 2023-10-21T04:11:07.618 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:11:07.618 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:11:09.318 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:11:09.318 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.311+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:11:09.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:11:09.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:11:09.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:11:09.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:11:09.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:11:09.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:11:09.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:11:09.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:09.312+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:11:17.620 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:11:17.621 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:17.621 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:17.622 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:18.565 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:11:18.565 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:11:18.567 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861478 2023-10-21T04:11:18.567 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:18.567 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848573237 ]] 2023-10-21T04:11:18.567 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848573237 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\5\7\3\2\3\7 ]] 2023-10-21T04:11:18.568 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:11:21.698 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:11:21.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.693+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:11:21.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:11:21.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:11:21.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.693+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:11:21.700 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.694+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:11:21.700 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:21.694+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:11:28.568 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:11:28.568 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:28.569 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:28.569 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:29.497 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:11:29.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:11:29.498 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861489 2023-10-21T04:11:29.498 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848573237 ]] 2023-10-21T04:11:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848573237 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\5\7\3\2\3\7 ]] 2023-10-21T04:11:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:11:39.501 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:11:39.501 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:39.501 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:39.502 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:39.661 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:11:39.662 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:11:39.662 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:11:39.662 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:11:39.662 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:11:39.662 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:11:39.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:11:39.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:11:39.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:11:39.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:39.657+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:11:40.445 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:11:40.445 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:11:40.446 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861500 2023-10-21T04:11:40.446 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:40.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848573237 ]] 2023-10-21T04:11:40.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848573237 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\5\7\3\2\3\7 ]] 2023-10-21T04:11:40.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:11:43.829 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:43.827+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:11:43.829 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:43.827+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:11:49.937 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:49.934+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:11:49.938 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:49.934+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:11:49.938 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:11:49.934+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:11:50.447 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:11:50.448 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:50.448 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:50.450 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:51.366 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:11:51.367 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:11:51.367 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861511 2023-10-21T04:11:51.367 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:51.368 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848573237 ]] 2023-10-21T04:11:51.368 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848573237 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\5\7\3\2\3\7 ]] 2023-10-21T04:11:51.368 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:11:58.282 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:58.280+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:11:58.283 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:58.281+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:11:58.283 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:58.281+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:11:58.283 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:58.281+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:11:58.284 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:58.281+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:11:58.284 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:11:58.281+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:12:01.368 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:12:01.370 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:01.370 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:01.370 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:02.298 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848573237 2023-10-21T04:12:02.298 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:12:02.299 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861522 2023-10-21T04:12:02.299 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:02.300 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848573237 ]] 2023-10-21T04:12:02.300 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848573237 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\5\7\3\2\3\7 ]] 2023-10-21T04:12:02.300 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:12:12.301 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:12:12.302 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:12.302 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:12.305 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:13.243 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:12:13.244 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:12:13.244 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861533 2023-10-21T04:12:13.244 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:13.245 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3960257379 ]] 2023-10-21T04:12:13.245 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3960257379 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\5\7\3\2\3\7 ]] 2023-10-21T04:12:13.245 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3960257379 2023-10-21T04:12:13.462 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:12:13.458+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:12:13.463 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:12:13.458+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:12:13.463 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:12:13.458+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:12:13.463 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:12:13.458+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:12:15.761 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3960257379 until 2023-10-21T05:12:14.956185+0000 (3600 sec) 2023-10-21T04:12:15.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:12:15.784 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3960257379 2023-10-21T04:12:16.546 INFO:tasks.workunit.client.0.smithi040.stderr:listed 10 entries 2023-10-21T04:12:16.568 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:12:16.568 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:12:26.570 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:12:26.570 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:26.571 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:26.571 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:27.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:12:27.500 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:12:27.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861547 2023-10-21T04:12:27.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:27.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3960257379 ]] 2023-10-21T04:12:27.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3960257379 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\9\6\0\2\5\7\3\7\9 ]] 2023-10-21T04:12:27.502 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:12:34.140 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:12:34.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:12:34.142 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:12:34.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:12:34.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:12:34.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:12:34.144 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:12:34.144 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:12:34.145 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:12:34.138+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:12:37.503 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:12:37.503 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:37.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:37.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:38.440 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:12:38.440 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:12:38.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861558 2023-10-21T04:12:38.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:38.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3960257379 ]] 2023-10-21T04:12:38.442 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3960257379 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\9\6\0\2\5\7\3\7\9 ]] 2023-10-21T04:12:38.442 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:12:48.443 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:12:48.443 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:48.444 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:48.449 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:49.378 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:12:49.379 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:12:49.379 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861569 2023-10-21T04:12:49.380 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:49.380 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3960257379 ]] 2023-10-21T04:12:49.380 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3960257379 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\9\6\0\2\5\7\3\7\9 ]] 2023-10-21T04:12:49.380 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:12:59.384 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:12:59.384 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:59.384 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:59.401 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.004+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.005+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288ef9ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.013+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.013+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890073a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.014+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.014+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890073300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.017+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.017+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890072a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048ac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.037+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.037+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.041+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.041+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.043+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.046+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.046+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.051+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.051+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.053+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.053+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048a780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.061+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.061+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.070+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.070+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048bb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.071+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.071+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048bd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.074+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.074+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.091 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.086+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.091 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.086+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.091 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.087+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.091 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.087+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.092 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.087+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.092 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:13:00.087+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.321 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:13:00.321 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:13:00.323 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861580 2023-10-21T04:13:00.323 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:00.324 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3960257379 ]] 2023-10-21T04:13:00.324 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3960257379 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\9\6\0\2\5\7\3\7\9 ]] 2023-10-21T04:13:00.324 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:13:09.690 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:09.687+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:13:09.691 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:09.687+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:13:09.691 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:09.687+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:13:09.691 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:09.687+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:13:10.327 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:13:10.327 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:10.327 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:10.328 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:11.250 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3960257379 2023-10-21T04:13:11.250 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:13:11.251 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861591 2023-10-21T04:13:11.251 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:11.251 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3960257379 ]] 2023-10-21T04:13:11.252 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3960257379 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\9\6\0\2\5\7\3\7\9 ]] 2023-10-21T04:13:11.252 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:13:12.536 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:12.534+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:13:12.538 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:12.535+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:13:12.539 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:12.535+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:13:15.477 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:15.475+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:13:15.478 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:15.475+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:13:15.478 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:15.475+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:13:15.478 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:15.475+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:13:17.591 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:17.587+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:13:17.592 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:17.587+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:13:17.592 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:13:17.587+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:13:21.253 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:13:21.253 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:21.256 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:21.257 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:22.180 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:13:22.181 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:13:22.183 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861602 2023-10-21T04:13:22.183 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:22.183 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:13:22.183 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\9\6\0\2\5\7\3\7\9 ]] 2023-10-21T04:13:22.183 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2300530352 2023-10-21T04:13:23.875 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2300530352 until 2023-10-21T05:13:23.096888+0000 (3600 sec) 2023-10-21T04:13:23.902 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:13:23.904 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2300530352 2023-10-21T04:13:24.648 INFO:tasks.workunit.client.0.smithi040.stderr:listed 11 entries 2023-10-21T04:13:24.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:13:24.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:13:34.672 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:13:34.673 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:34.675 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:34.675 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:35.437 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:13:35.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:13:35.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:13:35.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:13:35.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:13:35.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:13:35.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:13:35.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:13:35.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:13:35.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:35.434+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:13:35.600 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:13:35.600 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:13:35.601 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861615 2023-10-21T04:13:35.601 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:35.601 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:13:35.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:13:35.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:13:45.603 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:13:45.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:45.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:45.609 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:46.532 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:13:46.533 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:13:46.534 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861626 2023-10-21T04:13:46.534 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:46.534 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:13:46.534 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:13:46.534 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:13:49.769 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.766+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:13:49.771 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.769+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:13:49.772 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.769+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:13:49.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.773+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:13:49.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.773+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:13:49.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.773+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:13:49.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.773+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:13:49.776 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:49.773+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:13:50.611 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:13:50.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:13:50.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:13:50.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:13:50.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:13:50.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:13:50.613 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:13:50.613 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:13:50.608+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:13:56.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:13:56.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:56.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:56.540 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:57.439 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:13:57.440 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:13:57.440 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861637 2023-10-21T04:13:57.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:57.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:13:57.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:13:57.441 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:14:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.009+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.009+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.009+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.009+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.011+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.011+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.015+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.015+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.018+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.018+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.022+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.022+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.025+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.025+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.028+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.028+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.031+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.032+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.035+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.035+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.039+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.040+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.042+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.042+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.047+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.047+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.055+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.055+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.058+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.058+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.062+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.062+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3ec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.064+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.064+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3ee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.068+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.068+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3f000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.071+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.071+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.074+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:14:00.075+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3f400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:07.442 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:14:07.442 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:07.443 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:07.446 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:08.378 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:14:08.378 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:14:08.379 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861648 2023-10-21T04:14:08.379 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:08.379 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:14:08.379 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:14:08.380 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:14:15.424 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:15.423+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:14:15.425 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:15.423+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:14:15.425 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:15.423+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:14:15.425 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:15.423+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:14:18.380 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:14:18.383 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:18.383 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:18.384 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:18.853 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:18.851+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:14:19.326 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:14:19.326 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:14:19.328 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861659 2023-10-21T04:14:19.328 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:19.328 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:14:19.329 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:14:19.329 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:14:29.011 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:29.006+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:14:29.011 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:29.006+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:14:29.011 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:29.006+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:14:29.011 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:14:29.006+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:14:29.329 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:14:29.329 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:29.330 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:29.333 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:30.279 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2300530352 2023-10-21T04:14:30.279 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:14:30.280 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861670 2023-10-21T04:14:30.280 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:30.280 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2300530352 ]] 2023-10-21T04:14:30.280 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2300530352 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:14:30.281 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:14:40.282 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:14:40.282 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:40.283 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:40.289 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:41.202 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:14:41.203 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:14:41.204 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861681 2023-10-21T04:14:41.204 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:41.204 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/655843917 ]] 2023-10-21T04:14:41.205 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/655843917 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\3\0\0\5\3\0\3\5\2 ]] 2023-10-21T04:14:41.205 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/655843917 2023-10-21T04:14:42.791 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/655843917 until 2023-10-21T05:14:42.017349+0000 (3600 sec) 2023-10-21T04:14:42.811 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:14:42.812 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/655843917 2023-10-21T04:14:43.579 INFO:tasks.workunit.client.0.smithi040.stderr:listed 12 entries 2023-10-21T04:14:43.596 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:14:43.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:14:43.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.637+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:14:43.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.637+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:14:43.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.643+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:14:43.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.643+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:14:43.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.643+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:14:43.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.643+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:14:43.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.643+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:14:43.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:14:43.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:14:43.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:14:43.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:14:43.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:14:43.893 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:14:43.893 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:14:43.893 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:14:43.893 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:14:43.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:14:53.599 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:14:53.599 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:53.600 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:53.602 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:54.531 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:14:54.531 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:14:54.532 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861694 2023-10-21T04:14:54.532 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:54.532 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/655843917 ]] 2023-10-21T04:14:54.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/655843917 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\5\5\8\4\3\9\1\7 ]] 2023-10-21T04:14:54.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:15:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.005+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.005+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe3fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.006+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.006+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe3fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.015+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.015+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe3ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.016+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.016+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.022+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.022+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.027+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.027+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.032+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.032+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.038+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.039+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.039+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.043+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.048+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.048+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.051+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.051+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.055+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.059+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.065+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.066+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.069+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.069+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.073+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.073+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.076+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.077+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890309f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.080+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:15:00.080+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.534 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:15:04.535 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:04.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:04.538 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:05.472 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:15:05.473 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:15:05.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861705 2023-10-21T04:15:05.475 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:05.475 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/655843917 ]] 2023-10-21T04:15:05.475 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/655843917 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\5\5\8\4\3\9\1\7 ]] 2023-10-21T04:15:05.475 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:15:12.696 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:15:12.698 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:15:12.698 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:15:12.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:15:12.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:15:12.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:15:12.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:12.693+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:15:13.658 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:13.655+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:15:13.658 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:13.655+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:15:13.658 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:13.655+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:15:13.661 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:13.657+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:15:13.661 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:13.657+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:15:15.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:15:15.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:15.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:15.477 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:16.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:15:16.411 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:15:16.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861716 2023-10-21T04:15:16.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:16.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/655843917 ]] 2023-10-21T04:15:16.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/655843917 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\5\5\8\4\3\9\1\7 ]] 2023-10-21T04:15:16.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:15:26.413 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:15:26.413 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:26.414 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:26.414 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:27.351 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:15:27.352 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:15:27.353 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861727 2023-10-21T04:15:27.353 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:27.353 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/655843917 ]] 2023-10-21T04:15:27.353 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/655843917 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\5\5\8\4\3\9\1\7 ]] 2023-10-21T04:15:27.354 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:15:36.355 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:15:36.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:15:36.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:15:36.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:15:36.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:15:36.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:15:36.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:15:36.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:15:36.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:15:36.358 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:36.353+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:15:37.357 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:15:37.357 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:37.357 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:37.357 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:38.360 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/655843917 2023-10-21T04:15:38.361 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:15:38.362 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861738 2023-10-21T04:15:38.362 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:38.363 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/655843917 ]] 2023-10-21T04:15:38.363 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/655843917 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\5\5\8\4\3\9\1\7 ]] 2023-10-21T04:15:38.363 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:15:41.023 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:15:41.019+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:15:41.023 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:15:41.019+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:15:41.023 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:15:41.019+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:15:41.023 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:15:41.019+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:15:44.158 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:15:44.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:15:44.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:15:44.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:15:44.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:15:44.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:15:44.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:44.153+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:15:48.374 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:15:48.374 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:48.375 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:48.375 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:48.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:15:48.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:15:50.038 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:15:50.038 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:15:50.146 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861750 2023-10-21T04:15:50.146 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:50.146 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/771202222 ]] 2023-10-21T04:15:50.147 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/771202222 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\5\5\8\4\3\9\1\7 ]] 2023-10-21T04:15:50.147 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/771202222 2023-10-21T04:15:51.908 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/771202222 until 2023-10-21T05:15:51.143999+0000 (3600 sec) 2023-10-21T04:15:51.933 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:15:51.933 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/771202222 2023-10-21T04:15:52.686 INFO:tasks.workunit.client.0.smithi040.stderr:listed 13 entries 2023-10-21T04:15:52.704 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:15:52.704 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:16:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.003+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.003+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.008+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.008+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045aa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.014+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.014+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045ac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.019+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.019+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.025+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.025+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045b000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.030+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.030+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045b200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.033+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.033+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.038+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.038+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045b680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.043+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045bd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.046+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.046+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045bc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.049+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.049+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.053+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.054+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045b480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.064+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.064+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.065+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.065+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.067+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.067+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.070+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.070+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.074+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.074+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.077+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.077+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.080+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:16:00.080+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.374 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:00.371+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:16:00.375 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:00.371+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:00.375 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:00.371+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:16:02.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:16:02.719 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:02.719 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:02.719 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:03.685 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:16:03.685 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:16:03.687 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861763 2023-10-21T04:16:03.687 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:03.687 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/771202222 ]] 2023-10-21T04:16:03.687 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/771202222 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\7\1\2\0\2\2\2\2 ]] 2023-10-21T04:16:03.688 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:16:13.587 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:13.583+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:16:13.588 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:13.585+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:13.588 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:13.585+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:16:13.688 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:16:13.688 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:13.692 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:13.692 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:14.604 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:16:14.605 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:16:14.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861774 2023-10-21T04:16:14.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:14.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/771202222 ]] 2023-10-21T04:16:14.607 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/771202222 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\7\1\2\0\2\2\2\2 ]] 2023-10-21T04:16:14.607 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:16:23.122 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.119+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:16:23.124 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.121+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:16:23.124 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.121+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:16:23.125 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.123+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:16:23.125 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.123+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:16:23.125 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:16:23.126 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:23.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:16:23.420 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:23.417+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:24.610 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:16:24.610 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:24.611 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:24.612 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:25.535 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:16:25.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:16:25.537 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861785 2023-10-21T04:16:25.537 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:25.537 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/771202222 ]] 2023-10-21T04:16:25.537 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/771202222 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\7\1\2\0\2\2\2\2 ]] 2023-10-21T04:16:25.538 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:16:29.413 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.409+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:16:29.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.409+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:16:29.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.409+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:16:29.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.409+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:16:29.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.409+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:16:29.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.410+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:16:29.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.410+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:16:29.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.410+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:16:29.419 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.410+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:16:29.419 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:29.410+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:16:31.634 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.630+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:16:31.634 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.630+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:16:31.635 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.630+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:16:31.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:16:31.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:16:31.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:16:31.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:16:31.639 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:16:31.640 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:16:31.640 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:31.633+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:16:35.541 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:16:35.542 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:35.542 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:35.543 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:36.497 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:16:36.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:16:36.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861796 2023-10-21T04:16:36.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:36.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/771202222 ]] 2023-10-21T04:16:36.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/771202222 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\7\1\2\0\2\2\2\2 ]] 2023-10-21T04:16:36.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:16:41.024 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.019+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:16:41.025 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.019+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:16:41.025 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.022+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:16:41.026 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.024+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:16:41.026 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.024+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:16:41.026 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.024+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:16:41.027 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.024+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:16:41.027 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.024+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:16:41.027 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:41.024+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:16:41.462 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:41.459+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:16:41.462 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:41.459+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:41.652 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:41.649+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:43.078 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:43.072+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:45.936 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:45.934+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:16:45.936 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:45.934+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:45.937 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:45.934+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:16:46.502 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:16:46.502 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:46.502 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:46.505 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:47.308 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:47.304+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:16:47.309 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:47.304+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:47.309 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:47.304+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:16:47.463 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/771202222 2023-10-21T04:16:47.463 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:16:47.464 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861807 2023-10-21T04:16:47.464 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:47.465 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/771202222 ]] 2023-10-21T04:16:47.465 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/771202222 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\7\1\2\0\2\2\2\2 ]] 2023-10-21T04:16:47.465 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:16:52.128 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:16:52.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:16:52.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:16:52.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:16:52.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:16:52.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:16:52.130 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:16:52.130 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:16:52.130 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:16:52.130 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:16:52.123+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:16:56.619 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:56.617+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:16:56.620 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:56.617+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:16:56.620 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:16:56.617+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:16:57.466 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:16:57.467 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:57.467 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:57.468 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:58.373 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:16:58.374 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:16:58.375 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861818 2023-10-21T04:16:58.375 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:58.375 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1587307185 ]] 2023-10-21T04:16:58.375 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1587307185 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\7\1\2\0\2\2\2\2 ]] 2023-10-21T04:16:58.376 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1587307185 2023-10-21T04:17:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.012+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f768f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289048a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890072800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.021+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.021+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.021+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890073f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.021+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289048b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.024+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.024+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd59c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.030+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.030+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289045a780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.034+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.034+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289045bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.037+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.037+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fd58980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.041+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.041+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe82a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.045+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.045+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289028ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.048+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.048+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289045a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.052+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.053+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.066+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.067+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.067+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.067+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.068+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe3f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.068+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.069+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.069+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.077+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.077+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.080+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.080+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.084+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:17:00.084+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.853 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1587307185 until 2023-10-21T05:17:00.076934+0000 (3600 sec) 2023-10-21T04:17:00.875 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:17:00.876 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1587307185 2023-10-21T04:17:01.632 INFO:tasks.workunit.client.0.smithi040.stderr:listed 14 entries 2023-10-21T04:17:01.635 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:01.634+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:17:01.636 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:01.634+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:17:01.636 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:01.634+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:17:01.636 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:01.634+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:17:01.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:17:01.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:17:11.653 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:17:11.653 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:11.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:11.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:12.859 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:17:12.860 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:17:12.871 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861832 2023-10-21T04:17:12.871 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:12.871 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1587307185 ]] 2023-10-21T04:17:12.871 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1587307185 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\8\7\3\0\7\1\8\5 ]] 2023-10-21T04:17:12.872 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:17:14.285 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:14.283+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:17:14.286 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:14.283+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:17:14.286 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:14.283+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:17:17.269 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:17.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:17:17.270 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:17.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:17:17.270 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:17:17.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:17:22.181 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:17:22.182 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:17:22.182 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:17:22.182 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:17:22.182 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:17:22.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:17:22.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:17:22.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:17:22.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:17:22.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:22.179+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:17:22.876 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:17:22.876 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:22.876 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:22.876 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:23.795 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:17:23.795 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:17:23.796 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861843 2023-10-21T04:17:23.796 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:23.797 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1587307185 ]] 2023-10-21T04:17:23.797 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1587307185 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\8\7\3\0\7\1\8\5 ]] 2023-10-21T04:17:23.797 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:17:33.799 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:17:33.799 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:33.802 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:33.803 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:34.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:17:34.728 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:17:34.729 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861854 2023-10-21T04:17:34.729 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:34.729 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1587307185 ]] 2023-10-21T04:17:34.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1587307185 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\8\7\3\0\7\1\8\5 ]] 2023-10-21T04:17:34.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:17:44.734 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:17:44.734 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:44.734 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:44.735 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:45.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:17:45.668 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:17:45.669 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861865 2023-10-21T04:17:45.669 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:45.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1587307185 ]] 2023-10-21T04:17:45.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1587307185 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\8\7\3\0\7\1\8\5 ]] 2023-10-21T04:17:45.670 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:17:51.125 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.120+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:17:51.126 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.120+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:17:51.126 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:17:51.127 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:17:51.127 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:17:51.127 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:17:51.127 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:17:51.127 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:17:51.127 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:17:51.122+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:17:55.671 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:17:55.672 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:55.672 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:55.676 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:56.609 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1587307185 2023-10-21T04:17:56.609 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:17:56.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861876 2023-10-21T04:17:56.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:56.611 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1587307185 ]] 2023-10-21T04:17:56.611 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1587307185 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\8\7\3\0\7\1\8\5 ]] 2023-10-21T04:17:56.611 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:18:06.612 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:18:06.613 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:06.613 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:06.616 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:07.515 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:18:07.515 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:18:07.516 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861887 2023-10-21T04:18:07.516 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:07.516 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3201339169 ]] 2023-10-21T04:18:07.516 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3201339169 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\8\7\3\0\7\1\8\5 ]] 2023-10-21T04:18:07.516 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3201339169 2023-10-21T04:18:09.825 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3201339169 until 2023-10-21T05:18:09.046983+0000 (3600 sec) 2023-10-21T04:18:09.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:18:09.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3201339169 2023-10-21T04:18:10.613 INFO:tasks.workunit.client.0.smithi040.stderr:listed 15 entries 2023-10-21T04:18:10.632 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:18:10.632 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:18:11.562 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:18:11.563 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:18:11.563 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:18:11.563 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:18:11.563 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:18:11.563 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:18:11.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:18:11.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:18:11.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:18:11.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:11.560+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:18:20.635 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:18:20.635 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:20.635 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:20.636 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:21.574 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:18:21.575 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:18:21.576 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861901 2023-10-21T04:18:21.576 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:21.576 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3201339169 ]] 2023-10-21T04:18:21.577 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3201339169 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\2\0\1\3\3\9\1\6\9 ]] 2023-10-21T04:18:21.577 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:18:31.579 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:18:31.579 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:31.581 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:31.581 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:32.540 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:18:32.540 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:18:32.542 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861912 2023-10-21T04:18:32.542 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:32.542 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3201339169 ]] 2023-10-21T04:18:32.542 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3201339169 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\2\0\1\3\3\9\1\6\9 ]] 2023-10-21T04:18:32.543 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:18:42.546 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:42.547 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:42.547 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:18:42.547 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:43.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:18:43.460 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:18:43.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861923 2023-10-21T04:18:43.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:43.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3201339169 ]] 2023-10-21T04:18:43.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3201339169 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\2\0\1\3\3\9\1\6\9 ]] 2023-10-21T04:18:43.462 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:18:52.179 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:52.176+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:18:52.179 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:52.176+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:18:52.179 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:52.176+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:18:52.180 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:52.176+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:18:52.180 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:18:52.176+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:18:53.466 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:18:53.466 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:53.466 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:53.466 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:54.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:18:54.422 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:18:54.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861934 2023-10-21T04:18:54.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:54.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3201339169 ]] 2023-10-21T04:18:54.424 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3201339169 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\2\0\1\3\3\9\1\6\9 ]] 2023-10-21T04:18:54.424 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:19:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.003+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.003+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890309800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.014+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.014+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.014+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890372680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.014+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.015+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.015+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890373c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.023+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.023+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.023+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.023+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.028+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.028+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628903f9900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.033+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.033+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628903f9080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.038+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.038+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.042+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.042+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.046+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.046+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289048a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.050+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.050+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628903f9a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.055+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.055+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.059+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.059+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.063+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.063+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.067+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.067+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.072+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.072+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.078+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.078+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.081+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.081+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.084+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:19:00.084+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:04.166 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:04.162+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:19:04.167 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:04.162+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:19:04.428 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:19:04.428 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:04.428 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:04.428 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:05.383 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3201339169 2023-10-21T04:19:05.386 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:19:05.387 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861945 2023-10-21T04:19:05.387 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:05.388 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3201339169 ]] 2023-10-21T04:19:05.388 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3201339169 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\2\0\1\3\3\9\1\6\9 ]] 2023-10-21T04:19:05.388 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:19:15.394 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:19:15.394 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:15.394 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:15.404 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:16.366 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:19:16.366 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:19:16.370 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861956 2023-10-21T04:19:16.370 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:16.370 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:19:16.370 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\2\0\1\3\3\9\1\6\9 ]] 2023-10-21T04:19:16.371 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2568248431 2023-10-21T04:19:17.966 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2568248431 until 2023-10-21T05:19:17.179140+0000 (3600 sec) 2023-10-21T04:19:17.985 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:19:17.985 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2568248431 2023-10-21T04:19:18.730 INFO:tasks.workunit.client.0.smithi040.stderr:listed 16 entries 2023-10-21T04:19:18.749 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:19:18.749 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:19:28.751 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:19:28.752 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:28.752 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:28.753 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:29.842 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:19:29.843 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:19:29.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861969 2023-10-21T04:19:29.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:29.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:19:29.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:19:29.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:19:39.845 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:19:39.848 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:39.849 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:39.849 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:40.810 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:19:40.811 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:19:40.811 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861980 2023-10-21T04:19:40.811 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:40.811 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:19:40.812 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:19:40.812 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:19:41.293 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:41.290+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:19:41.293 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:41.290+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:19:41.294 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:41.290+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:19:49.365 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:19:49.362+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:19:49.370 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:19:49.368+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:19:49.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:19:49.368+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:19:49.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:19:49.368+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:19:49.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:19:49.369+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:19:50.813 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:19:50.813 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:50.814 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:50.814 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:51.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:19:51.753 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:19:51.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697861991 2023-10-21T04:19:51.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:51.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:19:51.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:19:51.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:19:51.897 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:51.895+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:19:51.897 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:51.895+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:19:51.897 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:51.895+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:19:51.897 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:51.895+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:19:58.361 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:58.359+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:19:58.362 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:58.359+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:19:58.362 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:58.359+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:19:58.362 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:19:58.359+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:20:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.007+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.007+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890373600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890372600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.014+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.014+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289045ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.018+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.018+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890073980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.022+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.022+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045be00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.028+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.028+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fab8580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.028+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.028+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.032+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.032+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f9080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.037+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.037+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe3f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.042+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.042+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fe3fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.045+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.045+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289048a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.048+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.048+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628903f8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.052+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.052+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628904d7380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.055+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.055+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628904d7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.059+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.059+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628904d7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.063+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.063+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628904d7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628904d7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890668080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.074+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.075+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890668280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.078+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:20:00.078+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890668480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:01.755 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:20:01.758 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:01.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:01.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:01.953 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:01.952+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:20:01.954 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:01.952+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:20:01.954 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:01.952+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:20:02.685 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:20:02.686 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:20:02.687 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862002 2023-10-21T04:20:02.687 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:02.688 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:20:02.688 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:20:02.688 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:20:06.316 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:06.314+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:20:06.316 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:06.314+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:20:06.316 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:06.314+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:20:06.654 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:06.647+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:20:06.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:06.649+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:20:06.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:06.649+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:20:06.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:06.653+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:20:12.689 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:20:12.690 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:12.691 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:12.691 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:13.531 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:20:13.532 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:20:13.532 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:20:13.532 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:20:13.532 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:20:13.532 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:20:13.533 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:20:13.533 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:20:13.533 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:20:13.533 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:13.528+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:20:13.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:20:13.652 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:20:13.652 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862013 2023-10-21T04:20:13.653 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:13.653 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:20:13.653 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:20:13.653 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:20:16.925 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.922+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:20:16.925 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.922+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:20:16.930 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:20:16.930 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:20:16.930 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:20:16.930 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:20:16.930 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:20:16.931 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:20:16.931 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:20:16.931 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:16.926+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:20:23.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:20:23.655 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:23.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:23.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:24.573 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2568248431 2023-10-21T04:20:24.574 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:20:24.574 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862024 2023-10-21T04:20:24.574 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:24.574 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2568248431 ]] 2023-10-21T04:20:24.574 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2568248431 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:20:24.575 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:20:34.575 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:20:34.575 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:34.576 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:34.587 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:35.518 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:20:35.519 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:20:35.520 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862035 2023-10-21T04:20:35.520 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:35.520 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/442953649 ]] 2023-10-21T04:20:35.520 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/442953649 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\8\2\4\8\4\3\1 ]] 2023-10-21T04:20:35.521 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/442953649 2023-10-21T04:20:37.927 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/442953649 until 2023-10-21T05:20:37.141245+0000 (3600 sec) 2023-10-21T04:20:37.951 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/442953649 2023-10-21T04:20:37.951 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:20:38.709 INFO:tasks.workunit.client.0.smithi040.stderr:listed 17 entries 2023-10-21T04:20:38.727 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:20:38.728 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:20:40.699 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:20:40.700 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:20:40.700 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:20:40.700 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:20:40.700 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:20:40.701 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:20:40.701 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:20:40.701 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:20:40.701 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:20:40.701 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:40.694+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:20:47.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.169+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:20:47.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.169+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:20:47.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.169+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:20:47.174 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:20:47.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:20:47.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:20:47.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:20:47.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:20:47.177 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:20:47.177 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:20:47.172+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:20:48.729 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:20:48.729 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:48.730 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:48.730 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:20:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:20:49.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862049 2023-10-21T04:20:49.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:49.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/442953649 ]] 2023-10-21T04:20:49.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/442953649 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\4\4\2\9\5\3\6\4\9 ]] 2023-10-21T04:20:49.643 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:20:56.128 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:56.123+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:20:59.489 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:59.487+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:20:59.490 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:59.487+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:20:59.490 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:59.487+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:20:59.490 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:20:59.487+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:20:59.644 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:20:59.644 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:59.645 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:59.648 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890308580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890373f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.019+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.026+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.026+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f768800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.034+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.034+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045a580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f9e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.044+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.044+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628904d7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.044+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.044+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628904d7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.046+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.046+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045b700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.050+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.050+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045be00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890668680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.059+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.059+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe3f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.063+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.063+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628904d6000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.071+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.071+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.071+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890668800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.071+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890668a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.074+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.074+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890668c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.082+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.083+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890669600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.083+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:21:00.083+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890669480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.663 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:21:00.663 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:21:00.664 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862060 2023-10-21T04:21:00.664 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:00.665 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/442953649 ]] 2023-10-21T04:21:00.665 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/442953649 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\4\4\2\9\5\3\6\4\9 ]] 2023-10-21T04:21:00.665 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:21:10.665 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:21:10.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:10.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:10.668 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:11.620 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:21:11.624 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:21:11.625 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862071 2023-10-21T04:21:11.625 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:11.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/442953649 ]] 2023-10-21T04:21:11.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/442953649 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\4\4\2\9\5\3\6\4\9 ]] 2023-10-21T04:21:11.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:21:21.631 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:21:21.631 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:21.631 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:21.632 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:22.555 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:21:22.555 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:21:22.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862082 2023-10-21T04:21:22.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:22.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/442953649 ]] 2023-10-21T04:21:22.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/442953649 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\4\4\2\9\5\3\6\4\9 ]] 2023-10-21T04:21:22.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:21:32.559 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:21:32.560 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:32.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:32.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:33.455 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/442953649 2023-10-21T04:21:33.458 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:21:33.458 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862093 2023-10-21T04:21:33.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:33.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/442953649 ]] 2023-10-21T04:21:33.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/442953649 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\4\4\2\9\5\3\6\4\9 ]] 2023-10-21T04:21:33.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:21:33.781 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:21:33.780+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:21:35.539 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:21:35.535+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:21:35.540 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:21:35.535+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:21:35.540 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:21:35.535+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:21:41.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:21:41.475 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:21:41.475 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:21:41.476 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:21:41.476 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:21:41.476 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:21:41.476 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:41.472+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:21:43.461 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:43.461 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:43.462 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:43.462 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:21:44.395 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:21:44.395 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:21:44.396 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:21:44.396 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:21:44.396 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:21:44.396 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:21:44.396 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:21:44.397 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:21:44.397 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:21:44.397 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:21:44.391+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:21:44.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:21:44.412 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:21:44.413 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862104 2023-10-21T04:21:44.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:44.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3705540092 ]] 2023-10-21T04:21:44.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3705540092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\4\4\2\9\5\3\6\4\9 ]] 2023-10-21T04:21:44.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3705540092 2023-10-21T04:21:46.901 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3705540092 until 2023-10-21T05:21:46.134345+0000 (3600 sec) 2023-10-21T04:21:46.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3705540092 2023-10-21T04:21:46.922 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:21:47.664 INFO:tasks.workunit.client.0.smithi040.stderr:listed 18 entries 2023-10-21T04:21:47.686 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:21:47.686 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:21:57.692 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:57.692 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:57.692 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:21:57.693 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:58.615 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:21:58.615 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:21:58.616 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862118 2023-10-21T04:21:58.617 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:58.617 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3705540092 ]] 2023-10-21T04:21:58.617 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3705540092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\5\5\4\0\0\9\2 ]] 2023-10-21T04:21:58.617 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:22:00.007 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.003+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.007 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.003+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fb8a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.009+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.009+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890073b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.010+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.010+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890309800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.017+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.017+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890073980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.018+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.018+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890309980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.022+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.022+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289045ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.026+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.026+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289028b880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.030+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.030+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f27a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.033+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.033+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890373680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.038+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.038+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f94c900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.041+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.041+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289028ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.045+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.045+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289028af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.048+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.048+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890669180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.051+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.051+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890668d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.055+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890669700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.058+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.058+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890669800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.062+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.062+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890669a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.066+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.066+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890669c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.069+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.069+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890669e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:22:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628906a2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:22:05.009 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.007+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:22:05.010 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.008+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:22:05.015 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.013+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:22:05.015 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.013+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:22:05.017 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.015+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:22:05.017 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.015+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:22:05.018 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.015+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:22:05.018 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.015+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:22:05.018 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:05.015+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:22:07.207 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.201+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:22:07.211 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.207+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:22:07.213 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:22:07.213 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:22:07.214 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:22:07.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:22:07.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:22:07.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:22:07.215 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:22:07.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:22:07.216 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:22:08.619 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:22:08.620 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:08.620 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:08.620 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:09.569 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:22:09.569 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:22:09.570 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862129 2023-10-21T04:22:09.570 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:09.570 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3705540092 ]] 2023-10-21T04:22:09.571 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3705540092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\5\5\4\0\0\9\2 ]] 2023-10-21T04:22:09.571 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:22:11.818 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.814+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:22:11.821 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:22:11.821 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:22:11.821 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:22:11.821 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:22:11.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:22:11.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:22:11.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:22:11.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:11.818+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:22:18.350 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:18.344+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:22:18.350 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:18.344+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:22:18.350 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:18.344+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:22:19.572 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:22:19.572 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:19.572 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:19.576 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:20.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:22:20.500 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:22:20.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862140 2023-10-21T04:22:20.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:20.502 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3705540092 ]] 2023-10-21T04:22:20.502 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3705540092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\5\5\4\0\0\9\2 ]] 2023-10-21T04:22:20.502 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:22:26.709 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:26.705+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:22:26.710 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:22:26.705+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:22:28.352 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:28.348+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:22:28.352 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:28.348+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:22:28.352 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:28.348+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:22:30.507 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:22:30.507 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:30.507 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:30.508 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:31.449 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:22:31.449 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:22:31.450 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862151 2023-10-21T04:22:31.450 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:31.450 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3705540092 ]] 2023-10-21T04:22:31.451 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3705540092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\5\5\4\0\0\9\2 ]] 2023-10-21T04:22:31.451 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:22:41.453 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:22:41.454 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:41.454 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:41.456 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:42.423 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3705540092 2023-10-21T04:22:42.423 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:22:42.424 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862162 2023-10-21T04:22:42.424 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:42.424 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3705540092 ]] 2023-10-21T04:22:42.425 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3705540092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\5\5\4\0\0\9\2 ]] 2023-10-21T04:22:42.425 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:22:47.384 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:47.379+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:22:47.385 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:47.379+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:22:47.385 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:22:47.379+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:22:52.426 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:22:52.426 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:52.427 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:52.428 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:53.324 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:22:53.324 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:22:53.325 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862173 2023-10-21T04:22:53.325 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:53.325 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2504071713 ]] 2023-10-21T04:22:53.325 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2504071713 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\0\5\5\4\0\0\9\2 ]] 2023-10-21T04:22:53.326 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2504071713 2023-10-21T04:22:55.869 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2504071713 until 2023-10-21T05:22:55.080726+0000 (3600 sec) 2023-10-21T04:22:55.888 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:22:55.888 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2504071713 2023-10-21T04:22:56.642 INFO:tasks.workunit.client.0.smithi040.stderr:listed 19 entries 2023-10-21T04:22:56.662 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:22:56.663 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:23:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.011+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.011+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.011+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628906a2480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.011+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a2680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.013+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.013+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a2280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.016+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.016+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628906a2880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.019+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.019+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628906a2a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.024+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.025+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628906a2c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.028+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.028+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.038+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.038+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.043+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.048+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.048+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.050+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.050+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a2f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.060+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.060+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.066+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.066+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.069+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.070+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.074+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.074+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.078+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.078+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.081+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.081+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.085+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:23:00.085+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:03.820 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:03.818+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:23:03.821 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:03.818+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:23:03.821 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:03.818+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:23:03.821 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:03.818+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:23:06.668 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:23:06.668 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:06.668 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:06.669 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:07.609 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:23:07.609 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:23:07.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862187 2023-10-21T04:23:07.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:07.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2504071713 ]] 2023-10-21T04:23:07.611 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2504071713 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\0\4\0\7\1\7\1\3 ]] 2023-10-21T04:23:07.611 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:23:08.079 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.072+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:23:08.079 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.077+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:23:08.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.080+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:23:08.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.080+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:23:08.083 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.081+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:23:08.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.084+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:23:08.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:23:08.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:23:08.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:08.084+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:23:17.614 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:23:17.615 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:17.615 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:17.615 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:18.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:23:18.557 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:23:18.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862198 2023-10-21T04:23:18.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:18.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2504071713 ]] 2023-10-21T04:23:18.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2504071713 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\0\4\0\7\1\7\1\3 ]] 2023-10-21T04:23:18.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:23:25.640 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.636+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:23:25.640 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.636+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:23:25.643 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.641+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:23:25.648 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:23:25.648 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:23:25.648 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:23:25.648 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:23:25.649 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:23:25.649 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:23:25.649 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:25.645+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:23:28.560 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:23:28.560 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:28.560 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:28.563 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:28.905 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:28.902+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:23:28.906 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:28.902+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:23:28.906 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:28.902+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:23:28.906 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:28.902+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:23:29.498 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:23:29.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:23:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862209 2023-10-21T04:23:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2504071713 ]] 2023-10-21T04:23:29.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2504071713 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\0\4\0\7\1\7\1\3 ]] 2023-10-21T04:23:29.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:23:30.372 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:30.370+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:23:30.373 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:30.370+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:23:30.373 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:30.370+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:23:32.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:23:32.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:23:32.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:23:32.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:23:32.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:23:32.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:23:32.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:23:32.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:23:32.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:32.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:23:34.566 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:34.564+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:23:39.502 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:23:39.503 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:39.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:39.505 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:40.451 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:23:40.451 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:23:40.452 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862220 2023-10-21T04:23:40.453 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:40.453 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2504071713 ]] 2023-10-21T04:23:40.453 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2504071713 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\0\4\0\7\1\7\1\3 ]] 2023-10-21T04:23:40.453 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:23:50.456 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:23:50.456 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:50.460 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:50.460 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:51.392 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2504071713 2023-10-21T04:23:51.392 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:23:51.393 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862231 2023-10-21T04:23:51.393 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:51.394 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2504071713 ]] 2023-10-21T04:23:51.394 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2504071713 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\0\4\0\7\1\7\1\3 ]] 2023-10-21T04:23:51.394 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:23:55.767 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:55.764+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:23:55.767 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:55.764+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:23:55.767 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:55.764+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:23:55.767 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:23:55.764+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:23:56.962 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:23:56.963 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:23:56.963 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:23:56.963 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:23:56.963 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:23:56.963 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:23:56.964 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:23:56.964 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:23:56.964 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:23:56.957+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:24:01.395 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:24:01.398 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:01.399 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:01.399 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:02.332 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:24:02.332 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:24:02.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862242 2023-10-21T04:24:02.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:02.335 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/746785310 ]] 2023-10-21T04:24:02.335 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/746785310 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\0\4\0\7\1\7\1\3 ]] 2023-10-21T04:24:02.335 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/746785310 2023-10-21T04:24:03.920 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:03.918+0000 7f4e35e0d640 -1 librbd::ImageWatcher: 0x5628908bc000 image watch failed: 94732210690048, (107) Transport endpoint is not connected 2023-10-21T04:24:03.921 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:03.918+0000 7f4e35e0d640 -1 librbd::Watcher: 0x5628908bc000 handle_error: handle=94732210690048: (107) Transport endpoint is not connected 2023-10-21T04:24:03.992 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:03.990+0000 7f4e35e0d640 -1 librbd::watcher::RewatchRequest: 0x56288f104780 handle_unwatch client blocklisted 2023-10-21T04:24:03.992 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:03.990+0000 7f4e35e0d640 -1 librbd::Watcher: 0x5628908bc000 handle_rewatch: client blocklisted 2023-10-21T04:24:04.849 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/746785310 until 2023-10-21T05:24:04.075673+0000 (3600 sec) 2023-10-21T04:24:04.871 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:24:04.872 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/746785310 2023-10-21T04:24:04.888 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:04.886+0000 7f4e3560c640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:04.889 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:04.886+0000 7f4e3560c640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5628904b96c0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:04.889 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:04.886+0000 7f4e3560c640 -1 librbd::io::AioCompletion: 0x562890831b80 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:05.190 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:05.187+0000 7f4e3560c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:05.190 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:05.187+0000 7f4e3560c640 -1 librbd::io::AioCompletion: 0x56288f7771e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:05.190 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:05.187+0000 7f4e3560c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5628908ea580 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:05.191 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:24:05.189+0000 7f4e3560c640 -1 librbd::managed_lock::ReleaseRequest: 0x56288f1e15c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:24:05.628 INFO:tasks.workunit.client.0.smithi040.stderr:listed 20 entries 2023-10-21T04:24:05.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:24:05.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:24:15.431 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:15.429+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:24:15.434 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:15.433+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:24:15.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:24:15.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:15.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:15.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:16.576 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:24:16.577 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:24:16.577 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862256 2023-10-21T04:24:16.578 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:16.578 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/746785310 ]] 2023-10-21T04:24:16.578 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/746785310 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\4\6\7\8\5\3\1\0 ]] 2023-10-21T04:24:16.578 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:24:24.029 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:24.026+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:24:24.029 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:24.026+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:24:24.029 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:24.026+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:24:24.030 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:24.026+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:24:26.579 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:24:26.580 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:26.580 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:26.582 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:27.526 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:24:27.527 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:24:27.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862267 2023-10-21T04:24:27.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:27.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/746785310 ]] 2023-10-21T04:24:27.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/746785310 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\4\6\7\8\5\3\1\0 ]] 2023-10-21T04:24:27.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:24:32.113 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.110+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:24:32.117 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.115+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:24:32.118 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.115+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:24:32.118 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.115+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:24:32.119 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.115+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:24:32.119 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.115+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:24:32.119 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:32.115+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:24:36.104 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:36.096+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:24:36.104 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:36.096+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:24:37.529 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:24:37.532 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:37.533 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:37.533 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:38.469 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:24:38.469 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:24:38.470 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862278 2023-10-21T04:24:38.470 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:38.470 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/746785310 ]] 2023-10-21T04:24:38.470 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/746785310 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\4\6\7\8\5\3\1\0 ]] 2023-10-21T04:24:38.471 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:24:45.516 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:24:45.514+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:24:45.517 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:24:45.514+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:24:45.517 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:24:45.514+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:24:48.472 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:48.472 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:48.474 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:48.474 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:24:49.395 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:24:49.395 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:24:49.395 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862289 2023-10-21T04:24:49.395 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:49.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/746785310 ]] 2023-10-21T04:24:49.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/746785310 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\4\6\7\8\5\3\1\0 ]] 2023-10-21T04:24:49.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:24:50.745 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:50.737+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:24:50.745 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:24:50.737+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:24:59.400 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:59.400 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:24:59.400 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:59.400 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:00.138 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.136+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:00.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.136+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:00.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.136+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:00.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.136+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:00.312 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/746785310 2023-10-21T04:25:00.312 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:25:00.313 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862300 2023-10-21T04:25:00.313 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:00.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/746785310 ]] 2023-10-21T04:25:00.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/746785310 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\4\6\7\8\5\3\1\0 ]] 2023-10-21T04:25:00.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:25:00.665 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:00.663+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:25:00.666 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:00.663+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:25:00.666 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:00.663+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:25:00.666 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:00.663+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:25:00.771 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.769+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:00.771 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.769+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:00.771 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.769+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:00.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:00.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:00.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:00.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:00.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:00.776 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:00.776 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:00.773+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:25:07.391 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:07.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:07.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:07.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:07.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:07.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:07.393 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:07.393 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:07.393 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:07.393 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:07.388+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:25:10.317 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:25:10.317 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:10.317 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:10.318 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:11.254 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:25:11.255 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:25:11.255 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862311 2023-10-21T04:25:11.255 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:11.256 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3096266244 ]] 2023-10-21T04:25:11.256 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3096266244 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\4\6\7\8\5\3\1\0 ]] 2023-10-21T04:25:11.256 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3096266244 2023-10-21T04:25:12.978 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3096266244 until 2023-10-21T05:25:12.195201+0000 (3600 sec) 2023-10-21T04:25:13.001 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:25:13.002 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3096266244 2023-10-21T04:25:13.768 INFO:tasks.workunit.client.0.smithi040.stderr:listed 21 entries 2023-10-21T04:25:13.790 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:25:13.790 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:25:16.427 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.425+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:16.431 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:16.432 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:16.432 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:16.432 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:16.432 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:16.432 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:16.433 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:16.433 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.429+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:25:16.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:16.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:16.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:16.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:16.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:16.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:16.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:16.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:16.862 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:16.857+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:20.943 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:20.936+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:25:20.944 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:20.936+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:25:20.944 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:20.936+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:25:23.793 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:25:23.793 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:23.793 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:23.793 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:23.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.810+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:23.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.810+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:23.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:23.816 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.811+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:23.816 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:23.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.816+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:23.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.816+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:23.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.816+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:23.822 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.816+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:23.823 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:23.816+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:25:24.766 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:25:24.769 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:25:24.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862324 2023-10-21T04:25:24.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:24.771 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3096266244 ]] 2023-10-21T04:25:24.771 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3096266244 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\9\6\2\6\6\2\4\4 ]] 2023-10-21T04:25:24.771 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:25:26.167 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:26.171 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:26.171 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:26.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:26.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:26.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:26.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:26.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:26.173 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:26.173 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:26.165+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:25:31.017 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:31.014+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:25:31.018 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:31.014+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:25:31.018 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:31.014+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:25:31.018 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:31.014+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:25:32.790 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:32.787+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:25:32.791 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:32.787+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:25:32.791 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:32.787+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:25:32.791 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:32.787+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:25:33.829 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:33.826+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:25:33.829 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:33.826+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:25:33.830 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:33.826+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:25:34.774 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:25:34.774 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:34.774 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:34.779 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:35.719 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:25:35.720 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:25:35.721 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862335 2023-10-21T04:25:35.721 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:35.721 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3096266244 ]] 2023-10-21T04:25:35.721 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3096266244 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\9\6\2\6\6\2\4\4 ]] 2023-10-21T04:25:35.721 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:25:37.447 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:25:37.445+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:25:45.723 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:25:45.723 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:45.725 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:45.725 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:46.666 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:25:46.667 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:25:46.667 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862346 2023-10-21T04:25:46.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:46.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3096266244 ]] 2023-10-21T04:25:46.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3096266244 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\9\6\2\6\6\2\4\4 ]] 2023-10-21T04:25:46.668 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:25:49.131 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:49.131 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:49.131 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:25:49.131 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:25:49.132 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:25:49.132 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:25:49.132 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:25:49.132 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:25:49.132 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:25:49.133 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:49.127+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:25:51.642 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:51.637+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:25:51.642 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:25:51.637+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:25:56.671 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:25:56.672 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:56.672 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:56.673 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:57.490 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:25:57.490 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:25:57.491 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862357 2023-10-21T04:25:57.491 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:57.491 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3096266244 ]] 2023-10-21T04:25:57.492 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3096266244 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\9\6\2\6\6\2\4\4 ]] 2023-10-21T04:25:57.492 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:26:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.010+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.010+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f27ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.010+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.010+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289048a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.012+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.012+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890072880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.016+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.016+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289045af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.019+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.019+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f0c4a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.022+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.022+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.025+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.025+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.029+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.029+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.032+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.032+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.035+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.035+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628904d6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.039+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.039+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890668d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.043+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.043+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890669500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.046+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.046+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628903f9980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.049+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.049+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f9b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.053+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.053+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628903f8500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.055+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.065+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.065+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.072+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:26:00.072+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:26:07.497 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:26:07.497 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:07.497 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:07.497 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:08.455 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3096266244 2023-10-21T04:26:08.456 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:26:08.456 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862368 2023-10-21T04:26:08.456 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:08.457 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3096266244 ]] 2023-10-21T04:26:08.457 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3096266244 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\9\6\2\6\6\2\4\4 ]] 2023-10-21T04:26:08.457 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:26:16.291 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:26:16.286+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:26:18.461 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:26:18.461 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:18.461 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:18.461 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:19.398 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:26:19.398 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:26:19.398 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862379 2023-10-21T04:26:19.399 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:19.399 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:26:19.399 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\9\6\2\6\6\2\4\4 ]] 2023-10-21T04:26:19.399 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3576288111 2023-10-21T04:26:20.950 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3576288111 until 2023-10-21T05:26:20.160903+0000 (3600 sec) 2023-10-21T04:26:20.971 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:26:20.971 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3576288111 2023-10-21T04:26:21.741 INFO:tasks.workunit.client.0.smithi040.stderr:listed 22 entries 2023-10-21T04:26:21.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:26:21.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:26:24.219 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:26:24.216+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:26:24.219 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:26:24.216+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:26:24.220 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:26:24.216+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:26:24.220 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:26:24.216+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:26:31.763 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:26:31.764 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:31.764 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:31.768 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:32.699 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:26:32.700 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:26:32.701 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862392 2023-10-21T04:26:32.701 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:32.702 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:26:32.702 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:26:32.702 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:26:42.703 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:26:42.705 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:42.706 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:42.706 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:43.654 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:26:43.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:26:43.655 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862403 2023-10-21T04:26:43.655 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:43.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:26:43.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:26:43.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:26:53.657 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:26:53.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:53.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:53.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:54.601 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:26:54.601 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:26:54.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862414 2023-10-21T04:26:54.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:54.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:26:54.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:26:54.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:27:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.012+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628907fa100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fa300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fa500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.023+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.023+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fa700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.026+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.026+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fa900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.031+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.031+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.039+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.039+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fb100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.042+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.042+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907faf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.048+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.048+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.053+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.053+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fb500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.057+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.057+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fb300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.061+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.061+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.064+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.064+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fb900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.068+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.068+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fbb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.075+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.076+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fbf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.079+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.079+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890800100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.082+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.082+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890800300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.085+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.088 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.086+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890800500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.092 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.089+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:00.092 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:27:00.090+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890800700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:27:04.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:27:04.605 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:04.605 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:04.609 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:05.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:27:05.559 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:27:05.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862425 2023-10-21T04:27:05.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:05.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:27:05.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:27:05.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:27:07.331 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:07.326+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:27:15.563 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:27:15.563 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:15.565 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:15.569 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:16.508 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:27:16.509 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:27:16.510 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862436 2023-10-21T04:27:16.510 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:16.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:27:16.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:27:16.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:27:16.619 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:27:16.619 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:27:16.619 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:27:16.620 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:27:16.620 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:27:16.620 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:27:16.620 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:27:16.621 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:27:16.621 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:27:16.621 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:16.616+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:27:20.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.156+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:27:20.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.156+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:27:20.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.156+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:27:20.159 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:27:20.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:27:20.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:27:20.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:27:20.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:27:20.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:27:20.161 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:20.157+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:27:21.159 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:27:21.158+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:27:21.159 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:27:21.158+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:27:21.160 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:27:21.158+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:27:21.860 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T04:27:21.858+0000 7f5619ccc640 -1 reset not still connected to 0x55da9e755ba0 2023-10-21T04:27:21.860 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T04:27:21.859+0000 7f5619ccc640 -1 reset not still connected to 0x55da9eafd520 2023-10-21T04:27:21.860 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-10-21T04:27:21.859+0000 7f5619ccc640 -1 reset not still connected to 0x55da9fa85a00 2023-10-21T04:27:26.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:27:26.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:26.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:26.515 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:27.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3576288111 2023-10-21T04:27:27.459 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:27:27.460 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862447 2023-10-21T04:27:27.460 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:27.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3576288111 ]] 2023-10-21T04:27:27.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3576288111 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:27:27.461 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:27:37.464 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:27:37.465 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:37.465 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:37.470 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:38.409 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:27:38.410 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:27:38.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862458 2023-10-21T04:27:38.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:38.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/764582085 ]] 2023-10-21T04:27:38.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/764582085 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\5\7\6\2\8\8\1\1\1 ]] 2023-10-21T04:27:38.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/764582085 2023-10-21T04:27:40.925 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/764582085 until 2023-10-21T05:27:40.138066+0000 (3600 sec) 2023-10-21T04:27:40.946 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:27:40.946 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/764582085 2023-10-21T04:27:41.676 INFO:tasks.workunit.client.0.smithi040.stderr:listed 23 entries 2023-10-21T04:27:41.699 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:27:41.700 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:27:46.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:46.581+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:27:46.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:46.586+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:27:46.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:46.586+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:27:46.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:46.586+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:27:46.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:27:46.586+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:27:50.653 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:27:50.648+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:27:50.653 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:27:50.648+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:27:51.701 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:27:51.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:51.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:51.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:52.644 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:27:52.644 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:27:52.646 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862472 2023-10-21T04:27:52.646 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:52.646 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/764582085 ]] 2023-10-21T04:27:52.646 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/764582085 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\6\4\5\8\2\0\8\5 ]] 2023-10-21T04:27:52.646 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:28:00.005 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.003+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.003+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fab9180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.006+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.006+0000 7f4e3660e640 -1 librbd::ImageState: 0x562890800880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.013+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.013+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890800c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.014+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.014+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890800e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.017+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.017+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.021+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.021+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.029+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.029+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.038+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.041+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.042+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.045+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.045+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081c000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.048+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.048+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081c200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.052+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.052+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081c400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.055+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081ca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.065+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.065+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081cc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:28:00.068+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:28:02.650 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:28:02.650 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:02.650 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:02.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:03.571 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:28:03.571 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:28:03.572 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862483 2023-10-21T04:28:03.572 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:03.572 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/764582085 ]] 2023-10-21T04:28:03.573 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/764582085 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\6\4\5\8\2\0\8\5 ]] 2023-10-21T04:28:03.573 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:28:04.441 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.437+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:28:04.443 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:28:04.443 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:28:04.443 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:28:04.444 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:28:04.444 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:28:04.444 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:28:04.444 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:28:04.444 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:04.440+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:28:07.451 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:07.446+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:28:07.451 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:07.446+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:28:07.451 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:07.446+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:28:07.452 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:07.446+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:28:13.574 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:28:13.574 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:13.575 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:13.577 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:14.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.250+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:28:14.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.250+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:28:14.262 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.252+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:28:14.262 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.252+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:28:14.262 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.252+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:28:14.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.258+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:28:14.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.258+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:28:14.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.258+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:28:14.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.258+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:28:14.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:14.258+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:28:14.531 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:28:14.532 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:28:14.532 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862494 2023-10-21T04:28:14.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:14.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/764582085 ]] 2023-10-21T04:28:14.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/764582085 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\6\4\5\8\2\0\8\5 ]] 2023-10-21T04:28:14.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:28:19.166 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:19.156+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:28:19.167 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:19.156+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:28:24.534 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:28:24.535 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:24.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:24.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:25.495 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:28:25.496 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:28:25.496 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862505 2023-10-21T04:28:25.496 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:25.497 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/764582085 ]] 2023-10-21T04:28:25.497 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/764582085 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\6\4\5\8\2\0\8\5 ]] 2023-10-21T04:28:25.497 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:28:26.939 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:26.936+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:28:26.940 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:26.936+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:28:26.940 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:28:26.936+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:28:35.253 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:28:35.253 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:28:35.254 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:28:35.254 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:28:35.254 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:28:35.254 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:28:35.255 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:28:35.255 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:28:35.255 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:28:35.255 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:35.251+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:28:35.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:28:35.499 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:35.499 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:35.500 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:36.409 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/764582085 2023-10-21T04:28:36.409 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:28:36.410 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862516 2023-10-21T04:28:36.410 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:36.410 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/764582085 ]] 2023-10-21T04:28:36.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/764582085 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\6\4\5\8\2\0\8\5 ]] 2023-10-21T04:28:36.411 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:28:45.649 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:28:45.645+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:28:46.415 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:28:46.415 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:46.415 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:46.415 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:47.369 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:28:47.371 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:28:47.371 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862527 2023-10-21T04:28:47.371 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:47.372 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1663690888 ]] 2023-10-21T04:28:47.372 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1663690888 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\7\6\4\5\8\2\0\8\5 ]] 2023-10-21T04:28:47.372 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1663690888 2023-10-21T04:28:49.901 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1663690888 until 2023-10-21T05:28:49.132737+0000 (3600 sec) 2023-10-21T04:28:49.923 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1663690888 2023-10-21T04:28:49.924 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:28:50.678 INFO:tasks.workunit.client.0.smithi040.stderr:listed 24 entries 2023-10-21T04:28:50.699 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:28:50.699 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:29:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.003+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.003+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289048b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.009+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.009+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe3f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.010+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.010+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.015+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.015+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fe3ee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.017+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.017+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a2180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.021+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.021+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628904d6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045b880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.039+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.039+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.042+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.042+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628904d6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.054 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.051+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.051+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.051+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.051+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.052+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.052+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.068+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081dc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.068+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.069+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.069+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d2200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.071+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:29:00.071+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d2400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:29:00.701 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:29:00.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:00.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:00.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:01.629 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:29:01.629 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:29:01.631 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862541 2023-10-21T04:29:01.631 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:01.631 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1663690888 ]] 2023-10-21T04:29:01.631 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1663690888 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\6\6\3\6\9\0\8\8\8 ]] 2023-10-21T04:29:01.631 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:29:02.114 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:02.110+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:29:02.114 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:02.110+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:29:11.634 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:29:11.635 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:11.635 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:11.636 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:12.579 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:29:12.580 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:29:12.580 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862552 2023-10-21T04:29:12.580 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:12.580 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1663690888 ]] 2023-10-21T04:29:12.581 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1663690888 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\6\6\3\6\9\0\8\8\8 ]] 2023-10-21T04:29:12.581 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:29:14.585 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:14.583+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:29:17.590 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:29:17.590 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:17.591 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:29:17.591 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:29:17.591 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:17.591 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:17.591 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:29:17.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:17.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:17.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:29:17.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:29:17.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:17.592 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:29:17.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:29:17.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:17.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:17.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:29:17.594 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:17.594 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:17.594 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:17.587+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:29:22.581 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:29:22.584 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:22.585 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:22.585 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:23.517 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:29:23.517 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:29:23.518 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862563 2023-10-21T04:29:23.519 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:23.519 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1663690888 ]] 2023-10-21T04:29:23.519 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1663690888 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\6\6\3\6\9\0\8\8\8 ]] 2023-10-21T04:29:23.519 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:29:23.739 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:23.733+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:29:23.739 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:23.733+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:23.739 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:23.733+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:23.739 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:23.733+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:23.740 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:23.733+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:27.147 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:27.145+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:29:27.148 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:27.145+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:29:27.148 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:27.145+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:29:27.148 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:29:27.145+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:29:27.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:29:27.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:27.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:29:27.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:29:27.929 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:27.929 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:27.929 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:29:27.929 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:27.929 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:27.924+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:28.659 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:28.656+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:28.664 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:28.661+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:28.664 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:28.661+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:29:28.665 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:28.661+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:28.665 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:28.661+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:28.665 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:28.661+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:29:30.518 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.514+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:30.521 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.519+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:29:30.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:29:30.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:30.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:30.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:29:30.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:30.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:30.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:30.526+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:29:33.520 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:29:33.521 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:33.521 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:33.526 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:34.472 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:29:34.472 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:29:34.473 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862574 2023-10-21T04:29:34.473 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:34.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1663690888 ]] 2023-10-21T04:29:34.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1663690888 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\6\6\3\6\9\0\8\8\8 ]] 2023-10-21T04:29:34.474 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:29:39.035 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.033+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:29:39.037 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.033+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:39.038 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:29:39.038 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:29:39.038 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:39.038 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:39.039 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:29:39.039 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:39.039 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:39.039 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:39.036+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:29:44.475 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:29:44.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:44.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:44.479 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:45.395 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1663690888 2023-10-21T04:29:45.395 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:29:45.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862585 2023-10-21T04:29:45.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:45.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1663690888 ]] 2023-10-21T04:29:45.396 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1663690888 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\6\6\3\6\9\0\8\8\8 ]] 2023-10-21T04:29:45.397 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:29:54.257 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.255+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:29:54.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.259+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:29:54.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.259+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:29:54.262 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.259+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:29:54.267 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.265+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:29:54.268 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.265+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:29:54.268 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:29:54.265+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:29:55.398 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:29:55.399 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:55.400 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:55.400 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:56.356 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:29:56.357 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:29:56.357 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862596 2023-10-21T04:29:56.357 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:56.357 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2750481391 ]] 2023-10-21T04:29:56.358 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2750481391 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\6\6\3\6\9\0\8\8\8 ]] 2023-10-21T04:29:56.358 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2750481391 2023-10-21T04:29:58.019 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2750481391 until 2023-10-21T05:29:57.244365+0000 (3600 sec) 2023-10-21T04:29:58.040 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:29:58.040 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2750481391 2023-10-21T04:29:58.775 INFO:tasks.workunit.client.0.smithi040.stderr:listed 25 entries 2023-10-21T04:29:58.794 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:29:58.794 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:30:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.004+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.004+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f94c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.007+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.007+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f769500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.010+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.010+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d2780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.014+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.014+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d2580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.017+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.017+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.021+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.021+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d3c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.027+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.027+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.034+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.034+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.039+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.039+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.043+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.046+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.046+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072ee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.050+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.050+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.054+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.054+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.063+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.065+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.065+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.068+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.068+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289072f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.070+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:00.072 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:30:00.070+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289072f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:30:07.847 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:07.841+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:07.847 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:07.841+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:07.848 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:07.841+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:08.796 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:30:08.796 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:08.796 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:08.797 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:09.717 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:30:09.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:30:09.719 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862609 2023-10-21T04:30:09.719 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:09.719 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2750481391 ]] 2023-10-21T04:30:09.720 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2750481391 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\5\0\4\8\1\3\9\1 ]] 2023-10-21T04:30:09.720 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:30:12.508 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:12.502+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:12.508 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:12.504+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:12.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:12.508+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:30:17.506 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.504+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:17.507 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.504+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:17.507 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.504+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:30:17.515 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:17.515 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:17.515 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:17.515 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:17.516 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:17.516 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:17.516 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:17.512+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:30:19.721 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:30:19.721 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:19.722 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:19.722 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:20.078 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:20.067+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:30:20.078 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:20.067+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:30:20.078 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:20.068+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:30:20.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:30:20.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:30:20.657 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862620 2023-10-21T04:30:20.657 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:20.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2750481391 ]] 2023-10-21T04:30:20.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2750481391 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\5\0\4\8\1\3\9\1 ]] 2023-10-21T04:30:20.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:30:20.934 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:20.928+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:30:20.935 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:20.928+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:30:21.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:21.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:21.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:30:21.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:21.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:21.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:21.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:21.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:21.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:21.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:21.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:30:26.093 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:26.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:26.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:30:26.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:26.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:26.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:26.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:26.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:26.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:26.090+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:27.899 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.896+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:27.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.896+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:27.903 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:30:27.903 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:27.903 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:27.903 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:27.904 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:27.904 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:27.904 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:27.904 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:27.900+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:30:30.659 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:30:30.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:30.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:30.661 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:31.622 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:30:31.623 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:30:31.623 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862631 2023-10-21T04:30:31.624 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:31.624 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2750481391 ]] 2023-10-21T04:30:31.624 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2750481391 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\5\0\4\8\1\3\9\1 ]] 2023-10-21T04:30:31.624 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:30:32.517 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.514+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:32.520 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.518+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:32.520 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.518+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:32.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.524+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:32.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.524+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:32.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:32.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:32.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:32.524+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:30:41.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:30:41.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:41.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:41.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:42.574 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:30:42.574 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:30:42.575 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862642 2023-10-21T04:30:42.575 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:42.575 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2750481391 ]] 2023-10-21T04:30:42.575 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2750481391 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\5\0\4\8\1\3\9\1 ]] 2023-10-21T04:30:42.575 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:30:42.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:42.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:42.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:30:42.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:42.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:42.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:42.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:42.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:42.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.934+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:42.939 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:42.937+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:30:47.813 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:30:47.813 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:30:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:30:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:30:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:30:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:30:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:30:47.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:30:47.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:30:47.811+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:30:52.576 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:30:52.577 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:52.577 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:52.581 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:53.535 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2750481391 2023-10-21T04:30:53.535 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:30:53.536 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862653 2023-10-21T04:30:53.536 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:53.536 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2750481391 ]] 2023-10-21T04:30:53.537 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2750481391 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\5\0\4\8\1\3\9\1 ]] 2023-10-21T04:30:53.537 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:30:55.585 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:55.583+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:30:55.586 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:55.583+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:30:55.586 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:55.583+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:30:55.586 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:30:55.583+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:31:03.541 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:31:03.542 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:03.542 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:03.544 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:04.476 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:31:04.477 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:31:04.478 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862664 2023-10-21T04:31:04.478 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:04.478 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:31:04.479 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\5\0\4\8\1\3\9\1 ]] 2023-10-21T04:31:04.479 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2564787531 2023-10-21T04:31:06.997 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2564787531 until 2023-10-21T05:31:06.236580+0000 (3600 sec) 2023-10-21T04:31:07.018 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:31:07.019 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2564787531 2023-10-21T04:31:07.781 INFO:tasks.workunit.client.0.smithi040.stderr:listed 26 entries 2023-10-21T04:31:07.800 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:31:07.801 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:31:17.802 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:31:17.803 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:17.803 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:17.804 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:18.750 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:31:18.751 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:31:18.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862678 2023-10-21T04:31:18.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:18.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:31:18.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:31:18.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:31:23.639 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:23.638+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:31:23.640 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:23.638+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:31:23.640 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:23.638+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:31:27.500 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:27.493+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:31:28.753 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:31:28.754 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:28.755 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:28.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:29.701 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:31:29.701 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:31:29.702 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862689 2023-10-21T04:31:29.703 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:29.703 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:31:29.703 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:31:29.703 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:31:32.694 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:32.691+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:31:34.025 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.022+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:31:34.026 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.022+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:31:34.026 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.022+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:31:34.027 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.024+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:31:34.027 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.024+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:31:34.030 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.026+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:31:34.030 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.027+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:31:34.030 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.027+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:31:34.030 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.027+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:31:34.030 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:34.027+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:31:34.798 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:34.795+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:31:34.798 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:34.795+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:31:34.799 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:34.795+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:31:34.799 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:34.795+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:31:36.930 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.929+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:31:36.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.933+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:31:36.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.933+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:31:36.937 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.933+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:31:36.946 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.944+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:31:36.946 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.944+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:31:36.946 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.944+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:31:36.946 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.944+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:31:36.946 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:31:36.944+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:31:39.704 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:31:39.705 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:39.705 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:39.716 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:40.648 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:31:40.649 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:31:40.650 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862700 2023-10-21T04:31:40.650 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:40.650 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:31:40.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:31:40.651 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:31:50.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:31:50.655 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:50.655 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:50.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:51.595 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:31:51.595 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:31:51.596 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862711 2023-10-21T04:31:51.596 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:51.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:31:51.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:31:51.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:31:57.822 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:57.818+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:31:58.448 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:58.443+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:31:58.449 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:58.443+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:31:58.730 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:58.727+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:31:58.730 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:31:58.727+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:32:00.005 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.003+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.003+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.008+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.008+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.014+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.014+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.020+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.020+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.026+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.026+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.032+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.032+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.038+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.044+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.044+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.048+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.048+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.054+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.054+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec2c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.062+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.062+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.062+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec2e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.062+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.066+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.066+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.076+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.076+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.080+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.080+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.084+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.084+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.092+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.092+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.092+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fec3080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:00.097 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:32:00.092+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f3d6000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:32:01.100 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:32:01.098+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:32:01.100 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:32:01.098+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:32:01.598 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:32:01.598 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:01.599 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:01.599 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:02.533 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:32:02.534 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:32:02.534 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862722 2023-10-21T04:32:02.535 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:02.535 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:32:02.535 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:32:02.535 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:32:12.536 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:32:12.539 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:12.540 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:12.540 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:13.466 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2564787531 2023-10-21T04:32:13.467 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:32:13.467 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862733 2023-10-21T04:32:13.467 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:13.467 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2564787531 ]] 2023-10-21T04:32:13.468 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2564787531 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:32:13.468 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:32:15.676 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:32:15.676 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:32:15.677 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:32:15.677 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:32:15.677 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:32:15.677 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:32:15.677 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:32:15.678 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:32:15.678 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:32:15.678 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:15.673+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:32:18.752 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:18.747+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:32:18.753 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:18.747+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:32:18.753 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:18.747+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:32:18.753 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:18.748+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:32:23.469 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:32:23.470 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:23.471 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:23.478 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:24.388 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:32:24.388 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:32:24.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862744 2023-10-21T04:32:24.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:24.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/660687942 ]] 2023-10-21T04:32:24.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/660687942 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\5\6\4\7\8\7\5\3\1 ]] 2023-10-21T04:32:24.390 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/660687942 2023-10-21T04:32:25.230 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.228+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:32:25.240 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.237+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:32:25.240 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.237+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:32:25.248 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.246+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:32:25.249 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.246+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:32:25.249 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.246+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:32:25.249 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.246+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:32:25.249 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:25.246+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:32:25.997 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/660687942 until 2023-10-21T05:32:25.207408+0000 (3600 sec) 2023-10-21T04:32:26.022 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:32:26.022 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/660687942 2023-10-21T04:32:26.782 INFO:tasks.workunit.client.0.smithi040.stderr:listed 27 entries 2023-10-21T04:32:26.803 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:32:26.803 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:32:36.808 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:32:36.808 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:36.809 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:36.809 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:37.723 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:32:37.723 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:32:37.724 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862757 2023-10-21T04:32:37.725 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:37.725 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/660687942 ]] 2023-10-21T04:32:37.725 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/660687942 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\6\0\6\8\7\9\4\2 ]] 2023-10-21T04:32:37.725 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:32:47.728 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:32:47.728 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:47.732 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:47.733 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:48.682 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:32:48.683 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:32:48.684 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862768 2023-10-21T04:32:48.684 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:48.684 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/660687942 ]] 2023-10-21T04:32:48.685 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/660687942 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\6\0\6\8\7\9\4\2 ]] 2023-10-21T04:32:48.685 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:32:56.890 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:32:56.891 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:32:56.891 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:32:56.891 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:32:56.891 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:32:56.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:32:56.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:32:56.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:32:56.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:32:56.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:32:56.888+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:32:58.685 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:32:58.687 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:58.688 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:58.688 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:59.622 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:32:59.622 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:32:59.623 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862779 2023-10-21T04:32:59.623 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:59.623 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/660687942 ]] 2023-10-21T04:32:59.623 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/660687942 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\6\0\6\8\7\9\4\2 ]] 2023-10-21T04:32:59.624 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:33:00.007 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.005+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.007 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.005+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289048a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.012+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.012+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628907fa580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.018+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.018+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.019+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.019+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628907fa800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.021+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.021+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.024+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.024+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.028+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.028+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.031+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.034+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.034+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.037+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.039 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.038+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.041+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.041+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.044+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.044+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.048+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.048+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.057+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.057+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.057+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.057+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.058+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.058+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.062+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.062+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.065+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.067 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.065+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.069+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.069+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289066e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.073+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:33:00.073+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289066e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:33:09.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:33:09.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:09.628 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:09.630 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:10.556 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:33:10.556 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:33:10.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862790 2023-10-21T04:33:10.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:10.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/660687942 ]] 2023-10-21T04:33:10.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/660687942 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\6\0\6\8\7\9\4\2 ]] 2023-10-21T04:33:10.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:33:18.384 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:33:18.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:33:18.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:33:18.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:33:18.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:33:18.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:33:18.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:33:18.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:33:18.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:33:18.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:18.382+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:33:20.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:33:20.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:20.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:20.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:21.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/660687942 2023-10-21T04:33:21.499 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:33:21.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862801 2023-10-21T04:33:21.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:21.500 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/660687942 ]] 2023-10-21T04:33:21.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/660687942 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\6\0\6\8\7\9\4\2 ]] 2023-10-21T04:33:21.501 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:33:31.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:33:31.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:31.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:31.504 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:32.406 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:33:32.406 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:33:32.407 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862812 2023-10-21T04:33:32.407 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:32.408 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848450024 ]] 2023-10-21T04:33:32.408 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848450024 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\6\6\0\6\8\7\9\4\2 ]] 2023-10-21T04:33:32.408 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2848450024 2023-10-21T04:33:33.959 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2848450024 until 2023-10-21T05:33:33.191890+0000 (3600 sec) 2023-10-21T04:33:33.976 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:33:33.976 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2848450024 2023-10-21T04:33:34.745 INFO:tasks.workunit.client.0.smithi040.stderr:listed 28 entries 2023-10-21T04:33:34.768 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:33:34.768 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:33:44.770 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:33:44.772 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:44.772 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:44.772 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:44.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:33:44.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:33:44.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:33:44.870 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:33:44.870 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:33:44.870 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:33:44.870 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:33:44.870 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:33:44.871 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:33:44.871 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:44.866+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:33:45.694 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:33:45.694 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:33:45.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862825 2023-10-21T04:33:45.696 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:45.696 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848450024 ]] 2023-10-21T04:33:45.696 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848450024 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\4\5\0\0\2\4 ]] 2023-10-21T04:33:45.696 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:33:45.986 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:45.982+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:33:45.987 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:45.982+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:33:46.288 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:46.275+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:33:46.461 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:46.459+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:33:47.292 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:47.289+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:33:47.292 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:47.289+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:33:47.292 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:33:47.289+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:33:51.689 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:33:51.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:33:51.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:33:51.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:33:51.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:33:51.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:33:51.691 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:33:51.691 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:33:51.691 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:33:51.691 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:51.685+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:33:54.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:33:54.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:33:54.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:33:54.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:33:54.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:33:54.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:33:54.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:33:54.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:33:54.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:33:54.928 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:33:54.925+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:33:55.697 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:33:55.698 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:55.698 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:55.698 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:56.625 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:33:56.625 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:33:56.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862836 2023-10-21T04:33:56.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:56.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848450024 ]] 2023-10-21T04:33:56.627 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848450024 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\4\5\0\0\2\4 ]] 2023-10-21T04:33:56.627 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:34:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.003+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.003+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.008 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.006+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.006+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.010+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.010+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.014+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.014+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.020+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.020+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.021+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.021+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628907fb200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.029+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.029+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890800100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.031+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.031+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.039+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.039+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.042+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.042+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.045+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.045+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.049+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.049+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.053+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.053+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.055+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.061+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.061+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.067+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.067+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:34:00.068+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:34:06.632 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:34:06.632 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:06.632 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:06.632 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:07.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:34:07.559 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:34:07.560 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862847 2023-10-21T04:34:07.560 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:07.560 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848450024 ]] 2023-10-21T04:34:07.560 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848450024 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\4\5\0\0\2\4 ]] 2023-10-21T04:34:07.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:34:12.599 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:34:12.595+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:34:12.599 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:34:12.595+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:34:12.599 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:34:12.595+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:34:17.161 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:34:17.157+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:34:17.161 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:34:17.157+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:34:17.566 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:34:17.566 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:17.566 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:17.566 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:18.520 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:34:18.521 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:34:18.522 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862858 2023-10-21T04:34:18.522 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:18.523 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848450024 ]] 2023-10-21T04:34:18.523 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848450024 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\4\5\0\0\2\4 ]] 2023-10-21T04:34:18.523 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:34:28.527 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:34:28.527 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:28.527 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:28.529 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:29.507 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2848450024 2023-10-21T04:34:29.508 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:34:29.509 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862869 2023-10-21T04:34:29.509 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:29.509 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2848450024 ]] 2023-10-21T04:34:29.509 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2848450024 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\4\5\0\0\2\4 ]] 2023-10-21T04:34:29.510 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:34:34.044 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.042+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:34:34.044 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.042+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:34:34.045 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.042+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:34:34.045 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.042+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:34:34.045 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.042+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:34:34.045 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.042+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:34:34.045 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.043+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:34:34.046 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.043+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:34:34.046 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.043+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:34:34.046 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:34.043+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:34:38.510 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:34:38.507+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:34:39.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:34:39.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:39.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:39.513 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:40.408 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:34:40.408 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:34:40.409 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862880 2023-10-21T04:34:40.409 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:40.410 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3333346210 ]] 2023-10-21T04:34:40.410 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3333346210 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\8\4\8\4\5\0\0\2\4 ]] 2023-10-21T04:34:40.410 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3333346210 2023-10-21T04:34:42.107 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3333346210 until 2023-10-21T05:34:41.318516+0000 (3600 sec) 2023-10-21T04:34:42.132 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:34:42.132 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3333346210 2023-10-21T04:34:42.864 INFO:tasks.workunit.client.0.smithi040.stderr:listed 29 entries 2023-10-21T04:34:42.891 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:34:42.891 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:34:51.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:34:51.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:34:51.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:34:51.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:34:51.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:34:51.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:34:51.322 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:34:51.322 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:51.317+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:34:52.897 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:52.898 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:52.898 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:34:52.899 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:53.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:34:53.831 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:34:53.831 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862893 2023-10-21T04:34:53.831 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:53.831 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3333346210 ]] 2023-10-21T04:34:53.832 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3333346210 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\3\3\3\4\6\2\1\0 ]] 2023-10-21T04:34:53.832 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:34:59.496 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.493+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:34:59.497 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.495+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:34:59.504 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.501+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:34:59.505 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.501+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:34:59.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.505+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:34:59.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.505+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:34:59.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:34:59.509 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.505+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:34:59.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:34:59.505+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:35:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.010+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.010+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.011+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.011+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.012+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.016+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.016+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.021+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.021+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.028+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.028+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.032+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.034 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.032+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.034+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.034+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d2f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.038+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.041+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.041+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.045+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.045+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.050 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.048+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.048+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.052+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.055 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.052+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.055+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.059+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.059+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.067+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.067+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.068+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.072+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:35:00.072+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:35:01.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:01.117+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:35:01.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:01.117+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:35:01.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:01.117+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:35:01.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:01.117+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:35:01.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:01.117+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:35:03.834 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:35:03.834 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:03.835 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:03.835 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:04.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:35:04.793 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:35:04.795 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862904 2023-10-21T04:35:04.795 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:04.795 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3333346210 ]] 2023-10-21T04:35:04.795 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3333346210 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\3\3\3\4\6\2\1\0 ]] 2023-10-21T04:35:04.796 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:35:06.599 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:06.596+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:35:06.600 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:06.596+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:35:14.797 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:35:14.798 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:14.798 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:14.798 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:14.961 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:14.954+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:35:14.962 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:14.954+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:35:14.962 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:14.954+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:35:15.758 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:35:15.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:35:15.760 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862915 2023-10-21T04:35:15.760 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:15.760 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3333346210 ]] 2023-10-21T04:35:15.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3333346210 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\3\3\3\4\6\2\1\0 ]] 2023-10-21T04:35:15.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:35:18.527 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:35:18.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:35:18.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:35:18.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:35:18.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:35:18.528 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:35:18.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:35:18.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:35:18.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:35:18.529 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:18.524+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:35:25.762 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:35:25.763 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:25.763 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:25.767 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:26.678 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:35:26.678 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:35:26.679 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862926 2023-10-21T04:35:26.679 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:26.679 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3333346210 ]] 2023-10-21T04:35:26.680 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3333346210 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\3\3\3\4\6\2\1\0 ]] 2023-10-21T04:35:26.680 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:35:26.956 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.953+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:35:26.956 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.953+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:35:26.956 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.953+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:35:26.957 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.954+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:35:26.957 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.954+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:35:26.958 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.956+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:35:26.958 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.956+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:35:26.959 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.956+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:35:26.959 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.956+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:35:26.959 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:26.956+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:35:31.137 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:31.135+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:35:31.138 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:31.135+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:35:31.139 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:31.135+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:35:31.139 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:35:31.135+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:35:36.681 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:35:36.681 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:36.682 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:36.682 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:37.639 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3333346210 2023-10-21T04:35:37.640 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:35:37.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862937 2023-10-21T04:35:37.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:37.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3333346210 ]] 2023-10-21T04:35:37.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3333346210 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\3\3\3\4\6\2\1\0 ]] 2023-10-21T04:35:37.643 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:35:47.446 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:47.444+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:35:47.446 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:47.444+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:35:47.446 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:47.444+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:35:47.646 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:35:47.646 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:47.646 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:47.646 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:48.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:35:48.563 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:35:48.563 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862948 2023-10-21T04:35:48.564 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:48.564 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3388017662 ]] 2023-10-21T04:35:48.564 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3388017662 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\3\3\3\4\6\2\1\0 ]] 2023-10-21T04:35:48.564 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3388017662 2023-10-21T04:35:51.062 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3388017662 until 2023-10-21T05:35:50.283900+0000 (3600 sec) 2023-10-21T04:35:51.082 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:35:51.083 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3388017662 2023-10-21T04:35:51.845 INFO:tasks.workunit.client.0.smithi040.stderr:listed 30 entries 2023-10-21T04:35:51.867 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:35:51.867 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:35:55.942 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:55.930+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:35:55.943 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:55.940+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:35:55.944 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:55.940+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:35:55.944 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:55.940+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:35:55.944 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:55.940+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:35:59.295 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.291+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:35:59.296 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.293+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:35:59.297 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.295+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:35:59.297 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.295+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:35:59.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.307+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:35:59.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.307+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:35:59.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.307+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:35:59.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:35:59.307+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:36:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.008+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.010 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.008+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.015+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.015+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.016+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.016+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.019+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.019+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.028+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.029 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.028+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.030+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.030+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.034+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.034+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.037+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.037+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.040+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.040+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.044+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.044+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.047+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.047+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f523e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.051+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.051+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.055+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.058 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.055+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.061+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.061+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.065 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.066+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.068 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.066+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.069+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.069+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050aa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.072+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.073+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050ac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.078+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.078+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.083+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:36:00.083+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:36:01.869 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:36:01.871 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:01.871 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:01.872 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:02.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:36:02.845 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:36:02.846 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862962 2023-10-21T04:36:02.847 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:02.847 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3388017662 ]] 2023-10-21T04:36:02.847 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3388017662 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\8\8\0\1\7\6\6\2 ]] 2023-10-21T04:36:02.847 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:36:04.138 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.129+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:36:04.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.129+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:36:04.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.129+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:36:04.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.138+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:36:04.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.138+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:36:04.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:36:04.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.138+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:36:04.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.138+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:36:04.370 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.363+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:36:04.370 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.363+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:36:04.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:36:04.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.367+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:36:04.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:36:04.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.369+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:36:04.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.369+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:36:04.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.369+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:36:04.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.369+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:36:04.373 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:04.369+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:36:11.262 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.257+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:36:11.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.257+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:36:11.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.257+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:36:11.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:36:11.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:36:11.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:36:11.265 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:36:11.265 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:36:11.265 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:36:11.265 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:11.261+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:36:12.280 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:36:12.277+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:36:12.280 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:36:12.277+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:36:12.280 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:36:12.277+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:36:12.847 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:36:12.879 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:12.880 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:12.880 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:13.782 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:36:13.782 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:36:13.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862973 2023-10-21T04:36:13.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:13.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3388017662 ]] 2023-10-21T04:36:13.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3388017662 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\8\8\0\1\7\6\6\2 ]] 2023-10-21T04:36:13.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:36:20.156 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:36:20.156 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:36:20.156 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:36:20.156 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:36:20.157 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:36:20.157 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:36:20.157 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:36:20.157 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:36:20.157 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:36:20.150+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:36:23.786 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:36:23.786 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:23.786 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:23.787 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:24.732 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:36:24.733 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:36:24.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862984 2023-10-21T04:36:24.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:24.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3388017662 ]] 2023-10-21T04:36:24.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3388017662 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\8\8\0\1\7\6\6\2 ]] 2023-10-21T04:36:24.734 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:36:34.736 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:36:34.736 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:34.736 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:34.737 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:35.693 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:36:35.693 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:36:35.694 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697862995 2023-10-21T04:36:35.694 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:35.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3388017662 ]] 2023-10-21T04:36:35.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3388017662 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\8\8\0\1\7\6\6\2 ]] 2023-10-21T04:36:35.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:36:45.696 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:36:45.696 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:45.701 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:45.702 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:46.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3388017662 2023-10-21T04:36:46.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:36:46.657 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863006 2023-10-21T04:36:46.657 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:46.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3388017662 ]] 2023-10-21T04:36:46.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3388017662 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\8\8\0\1\7\6\6\2 ]] 2023-10-21T04:36:46.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:36:52.195 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:36:52.189+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:36:56.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:36:56.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:56.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:56.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:57.604 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:36:57.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:36:57.605 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863017 2023-10-21T04:36:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1528643798 ]] 2023-10-21T04:36:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1528643798 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\3\8\8\0\1\7\6\6\2 ]] 2023-10-21T04:36:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1528643798 2023-10-21T04:37:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.003+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.006 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.004+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.009+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.011 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.009+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.012+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a2d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.016+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.016+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890668b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.020+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.020+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.029+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.029+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.035+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.038 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.035+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628907fa900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.038+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.040 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.038+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289050b480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.042 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1528643798 until 2023-10-21T05:36:59.246607+0000 (3600 sec) 2023-10-21T04:37:00.046 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.043+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.053+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.056 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.053+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.056+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.056+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.057+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289050b700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.061+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.061+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.064+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.064+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.071 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:37:00.072 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1528643798 2023-10-21T04:37:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.073+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.073+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.073+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.074+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.075+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.078 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.075+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890373a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.078+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.079+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.082+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:37:00.082+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:37:00.815 INFO:tasks.workunit.client.0.smithi040.stderr:listed 31 entries 2023-10-21T04:37:00.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:37:00.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:37:04.343 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:04.343 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:04.343 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:04.344 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:04.344 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:04.344 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:04.344 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:04.345 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:04.345 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:04.345 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:04.340+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:37:10.847 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:37:10.848 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:10.848 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:10.852 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:11.788 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:37:11.788 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:37:11.789 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863031 2023-10-21T04:37:11.789 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:11.790 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1528643798 ]] 2023-10-21T04:37:11.790 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1528643798 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\2\8\6\4\3\7\9\8 ]] 2023-10-21T04:37:11.790 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:37:21.759 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:37:21.758+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:37:21.760 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:37:21.758+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:37:21.790 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:37:21.790 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:21.790 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:21.791 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:22.750 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:37:22.750 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:37:22.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863042 2023-10-21T04:37:22.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:22.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1528643798 ]] 2023-10-21T04:37:22.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1528643798 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\2\8\6\4\3\7\9\8 ]] 2023-10-21T04:37:22.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:37:22.843 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.835+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:22.843 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:22.843 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:22.843 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:22.844 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:22.844 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:22.844 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:22.844 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:22.844 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:22.838+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:23.815 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:37:23.812+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:37:27.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.180+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:27.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.180+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:27.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.180+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:27.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.180+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:27.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.180+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:27.184 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.180+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:27.184 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.183+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:27.185 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:27.185 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:27.186 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:27.184+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:37:32.753 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:37:32.754 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:32.754 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:32.755 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:33.547 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:37:33.547 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:37:33.548 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863053 2023-10-21T04:37:33.548 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:33.548 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1528643798 ]] 2023-10-21T04:37:33.549 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1528643798 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\2\8\6\4\3\7\9\8 ]] 2023-10-21T04:37:33.549 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:37:43.107 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:43.105+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:43.107 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:43.105+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:43.112 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:43.109+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:43.112 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:43.109+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:43.112 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:43.109+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:43.122 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:43.120+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:37:43.550 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:43.550 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:43.551 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:43.556 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:37:44.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:37:44.528 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:37:44.529 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863064 2023-10-21T04:37:44.529 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:44.529 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1528643798 ]] 2023-10-21T04:37:44.529 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1528643798 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\2\8\6\4\3\7\9\8 ]] 2023-10-21T04:37:44.529 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:37:46.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.972+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:46.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.972+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:46.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.972+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:46.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.972+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:46.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.972+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:46.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.972+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:46.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.980+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:46.983 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.980+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:46.983 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.980+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:46.983 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:46.980+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:37:48.484 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:48.484 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:48.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:48.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:48.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:48.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:48.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:48.486 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:48.486 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:48.480+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:50.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.468+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:50.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.468+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:50.471 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.468+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:50.473 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.471+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:50.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.471+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:50.484 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.477+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:50.484 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.477+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:50.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.477+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:50.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.477+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:50.485 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:50.477+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:37:54.530 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:37:54.531 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:54.533 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:54.534 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:55.458 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1528643798 2023-10-21T04:37:55.458 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:37:55.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863075 2023-10-21T04:37:55.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:55.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1528643798 ]] 2023-10-21T04:37:55.459 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1528643798 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\2\8\6\4\3\7\9\8 ]] 2023-10-21T04:37:55.460 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:37:57.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:37:57.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:37:57.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:37:57.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:37:57.645 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:37:57.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:37:57.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:37:57.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:37:57.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:37:57.646 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:37:57.641+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:38:02.212 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:02.209+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:38:05.464 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:38:05.464 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:05.464 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:05.477 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:05.561 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:38:05.558+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:38:05.561 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:38:05.558+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:38:05.561 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:38:05.558+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:38:05.562 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:38:05.558+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:38:06.420 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:38:06.420 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:38:06.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863086 2023-10-21T04:38:06.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:06.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:38:06.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\5\2\8\6\4\3\7\9\8 ]] 2023-10-21T04:38:06.421 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3788273500 2023-10-21T04:38:08.142 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3788273500 until 2023-10-21T05:38:07.356737+0000 (3600 sec) 2023-10-21T04:38:08.167 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:38:08.167 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3788273500 2023-10-21T04:38:08.921 INFO:tasks.workunit.client.0.smithi040.stderr:listed 32 entries 2023-10-21T04:38:08.943 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:38:08.944 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:38:14.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:38:14.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:38:14.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:38:14.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:38:14.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:38:14.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:38:14.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:38:14.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:38:14.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:38:14.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:14.966+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:38:18.945 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:38:18.946 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:18.948 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:18.949 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:19.891 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:38:19.891 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:38:19.893 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863099 2023-10-21T04:38:19.893 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:19.893 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:38:19.893 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:38:19.893 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:38:29.896 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:38:29.896 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:29.897 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:29.897 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:30.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:38:30.822 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:38:30.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863110 2023-10-21T04:38:30.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:30.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:38:30.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:38:30.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:38:31.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:31.303+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:38:31.312 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:31.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:38:31.313 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:31.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:38:39.469 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:38:39.469 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:38:39.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:38:39.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:38:39.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:38:39.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:38:39.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:38:39.471 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:38:39.471 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:38:39.471 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:39.467+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:38:40.824 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:38:40.824 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:40.824 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:40.829 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:41.751 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:38:41.752 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:38:41.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863121 2023-10-21T04:38:41.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:41.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:38:41.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:38:41.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:38:46.307 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:38:46.308 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:38:46.308 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:38:46.308 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:38:46.308 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:38:46.308 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:38:46.309 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:38:46.309 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:38:46.309 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:38:46.309 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:46.305+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:38:51.547 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:51.543+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:38:51.550 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:51.543+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:38:51.550 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:38:51.543+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:38:51.758 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:38:51.758 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:51.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:51.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:52.693 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:38:52.694 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:38:52.694 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863132 2023-10-21T04:38:52.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:52.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:38:52.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:38:52.695 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:39:00.007 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.004+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.007 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.004+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.006+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.009 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.006+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.010+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.010+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d2080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.015+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.018 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.015+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.018+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.018+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.024+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.024+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628907fa600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.025+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.025+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.028+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.033 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.029+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289045bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.032+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.032+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.040+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.043 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.040+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628907fa880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.040+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.040+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289066e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.048 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.046+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.046+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.051+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.051+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890372980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.063 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.061+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.061+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.064+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.064+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.068+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.070 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.068+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.072+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.072+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.077+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.081 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.077+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.081+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:39:00.081+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050ba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:39:02.698 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:39:02.698 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:02.698 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:02.699 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:03.640 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:39:03.640 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:39:03.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863143 2023-10-21T04:39:03.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:03.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:39:03.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:39:03.642 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:39:13.644 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:39:13.645 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:13.645 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:13.647 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:14.585 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3788273500 2023-10-21T04:39:14.585 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:39:14.586 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863154 2023-10-21T04:39:14.586 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:14.586 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3788273500 ]] 2023-10-21T04:39:14.587 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3788273500 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:39:14.587 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:39:24.591 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:39:24.591 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:24.591 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:24.592 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:25.523 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:39:25.524 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:39:25.524 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863165 2023-10-21T04:39:25.525 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:25.525 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2732523112 ]] 2023-10-21T04:39:25.525 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2732523112 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\7\8\8\2\7\3\5\0\0 ]] 2023-10-21T04:39:25.525 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2732523112 2023-10-21T04:39:27.122 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2732523112 until 2023-10-21T05:39:26.346114+0000 (3600 sec) 2023-10-21T04:39:27.149 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:39:27.149 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2732523112 2023-10-21T04:39:27.929 INFO:tasks.workunit.client.0.smithi040.stderr:listed 33 entries 2023-10-21T04:39:27.949 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:39:27.950 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:39:31.329 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:39:31.330 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:39:31.330 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:39:31.330 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:39:31.330 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:39:31.331 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:39:31.331 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:39:31.331 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:39:31.331 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:31.326+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:39:37.951 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:39:37.952 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:37.952 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:37.952 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:38.907 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:39:38.911 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:39:38.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863178 2023-10-21T04:39:38.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:38.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2732523112 ]] 2023-10-21T04:39:38.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2732523112 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\3\2\5\2\3\1\1\2 ]] 2023-10-21T04:39:38.913 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:39:41.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:39:41.980 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:39:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:39:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:39:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:39:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:39:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:39:41.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:39:41.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:39:42.442 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.435+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:39:42.442 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.435+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:39:42.442 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.435+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:39:42.450 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:39:42.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:39:42.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:39:42.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:39:42.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:39:42.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:39:42.452 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:39:42.447+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:39:48.914 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:39:48.916 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:48.917 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:48.917 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:49.859 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:39:49.860 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:39:49.860 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863189 2023-10-21T04:39:49.861 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:49.861 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2732523112 ]] 2023-10-21T04:39:49.861 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2732523112 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\3\2\5\2\3\1\1\2 ]] 2023-10-21T04:39:49.861 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:39:51.360 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:39:51.352+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:39:59.866 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:39:59.866 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:59.866 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:59.867 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.013+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.013+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289081de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.014+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.014+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.014+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f768080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.014+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f78f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.021 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.015+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.015+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f3d7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.021+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.021+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289050bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.025+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.025+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289050bd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.031+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.032 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.031+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f522880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.036+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.037 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.036+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289050bc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.041+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.042 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.041+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628907fac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.046+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.047 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.046+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f523c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.052 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.051+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.051+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.056+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.057 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.056+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.060+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.060+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.065+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.066 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.065+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.073 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.071+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.072+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.075 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.074+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.074+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.079+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.079+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f524c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.085+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.085+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f524e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.091 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.090+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.092 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.090+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f525080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.095+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.096 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:40:00.095+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f525a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:40:00.820 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:40:00.821 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:40:00.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863200 2023-10-21T04:40:00.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:00.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2732523112 ]] 2023-10-21T04:40:00.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2732523112 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\3\2\5\2\3\1\1\2 ]] 2023-10-21T04:40:00.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:40:10.825 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:40:10.825 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:10.826 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:10.827 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:10.857 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:10.856+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:40:11.432 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:11.428+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:40:11.908 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:40:11.908 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:40:11.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863211 2023-10-21T04:40:11.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:11.913 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2732523112 ]] 2023-10-21T04:40:11.913 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2732523112 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\3\2\5\2\3\1\1\2 ]] 2023-10-21T04:40:11.913 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:40:16.086 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:40:16.086 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:40:16.086 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:40:16.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:40:16.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:40:16.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:40:16.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:16.084+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:40:21.914 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:40:21.915 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:21.915 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:21.915 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:22.899 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2732523112 2023-10-21T04:40:22.899 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:40:22.900 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863222 2023-10-21T04:40:22.900 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:22.900 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2732523112 ]] 2023-10-21T04:40:22.900 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2732523112 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\3\2\5\2\3\1\1\2 ]] 2023-10-21T04:40:22.900 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:40:32.902 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:40:32.902 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:32.905 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:32.906 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:33.865 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:40:33.865 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:40:33.866 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863233 2023-10-21T04:40:33.866 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:33.867 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3636778574 ]] 2023-10-21T04:40:33.867 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3636778574 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\7\3\2\5\2\3\1\1\2 ]] 2023-10-21T04:40:33.867 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3636778574 2023-10-21T04:40:34.415 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:40:34.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:40:34.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:40:34.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:40:34.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:40:34.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:40:34.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:40:34.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:40:34.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:34.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:40:36.134 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3636778574 until 2023-10-21T05:40:35.343152+0000 (3600 sec) 2023-10-21T04:40:36.154 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:40:36.155 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3636778574 2023-10-21T04:40:36.903 INFO:tasks.workunit.client.0.smithi040.stderr:listed 34 entries 2023-10-21T04:40:36.925 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:40:36.925 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:40:39.112 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:39.110+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:40:39.124 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:40:39.118+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:40:40.897 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:40.895+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:40:42.242 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:42.239+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:40:42.242 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:42.239+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:40:42.243 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:42.239+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:40:46.273 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:46.266+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:40:46.274 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:46.266+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:40:46.274 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:40:46.266+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:40:46.928 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:40:46.929 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:46.929 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:46.929 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:47.836 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:40:47.836 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:40:47.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863247 2023-10-21T04:40:47.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:47.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3636778574 ]] 2023-10-21T04:40:47.838 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3636778574 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\6\3\6\7\7\8\5\7\4 ]] 2023-10-21T04:40:47.839 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:40:57.840 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:40:57.840 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:57.841 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:57.841 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:58.791 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:40:58.792 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:40:58.792 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863258 2023-10-21T04:40:58.792 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:58.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3636778574 ]] 2023-10-21T04:40:58.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3636778574 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\6\3\6\7\7\8\5\7\4 ]] 2023-10-21T04:40:58.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:41:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.009+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.012 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.009+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289081c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.009+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.013 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.009+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fa900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628906d3b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.020+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.020+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.020+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628907fb900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.020+0000 7f4e3560c640 -1 librbd::ImageState: 0x5628906d3c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.025+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.025+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f3d2a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.028+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.028+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f3d2d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.032+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.032+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f3d2800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.082 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.040+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.040+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289050a780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.043+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.083 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.044+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289050ac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.048+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.048+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289050b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.054+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.054+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f3d3580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.058+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.058+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289072e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.061+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.061+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890800180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.065+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.065+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289050b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.069+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.069+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890669200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.072+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.072+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f78e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.076+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.076+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289066ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.087 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.082+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.088 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.082+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f525980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.089 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.086+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:00.089 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:41:00.087+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f525800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:41:02.309 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:02.307+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:41:02.309 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:02.307+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:41:08.795 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:41:08.797 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:08.798 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:08.798 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:09.759 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:41:09.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:41:09.760 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863269 2023-10-21T04:41:09.760 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:09.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3636778574 ]] 2023-10-21T04:41:09.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3636778574 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\6\3\6\7\7\8\5\7\4 ]] 2023-10-21T04:41:09.761 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:41:19.762 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:41:19.765 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:19.766 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:19.766 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:20.672 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:41:20.672 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:41:20.673 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863280 2023-10-21T04:41:20.674 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:20.674 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3636778574 ]] 2023-10-21T04:41:20.674 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3636778574 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\6\3\6\7\7\8\5\7\4 ]] 2023-10-21T04:41:20.674 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:41:22.248 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:22.246+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:41:23.970 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:23.968+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:41:23.970 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:23.968+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:41:23.970 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:23.968+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:41:23.971 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:23.968+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:41:30.676 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:41:30.676 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:30.677 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:30.679 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:31.635 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3636778574 2023-10-21T04:41:31.636 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:41:31.637 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863291 2023-10-21T04:41:31.637 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:31.638 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3636778574 ]] 2023-10-21T04:41:31.638 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3636778574 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\6\3\6\7\7\8\5\7\4 ]] 2023-10-21T04:41:31.638 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:41:32.038 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:32.037+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:41:32.039 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:32.037+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:41:34.060 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:34.058+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:41:34.061 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:34.058+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:41:34.061 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:34.058+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:41:34.061 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:41:34.058+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:41:38.984 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.979+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:41:38.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.979+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:41:38.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.979+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:41:38.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.979+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:41:38.985 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.979+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:41:38.986 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.979+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:41:38.986 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.981+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:41:38.986 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.981+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:41:38.986 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.981+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:41:38.987 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:38.981+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:41:41.638 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:41:41.639 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:41.641 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:41.642 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:42.561 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:41:42.561 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:41:42.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863302 2023-10-21T04:41:42.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:42.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1728514186 ]] 2023-10-21T04:41:42.563 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1728514186 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\6\3\6\7\7\8\5\7\4 ]] 2023-10-21T04:41:42.563 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/1728514186 2023-10-21T04:41:44.114 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/1728514186 until 2023-10-21T05:41:43.338659+0000 (3600 sec) 2023-10-21T04:41:44.140 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:41:44.141 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/1728514186 2023-10-21T04:41:44.884 INFO:tasks.workunit.client.0.smithi040.stderr:listed 35 entries 2023-10-21T04:41:44.906 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:41:44.906 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:41:52.458 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:41:52.459 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:41:52.459 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:41:52.459 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:41:52.459 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:41:52.460 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:41:52.460 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:41:52.460 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:41:52.460 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:41:52.460 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:41:52.456+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:41:54.911 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:54.911 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:41:54.911 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:54.912 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:55.774 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:41:55.774 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:41:55.775 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863315 2023-10-21T04:41:55.775 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:55.775 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1728514186 ]] 2023-10-21T04:41:55.776 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1728514186 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\7\2\8\5\1\4\1\8\6 ]] 2023-10-21T04:41:55.776 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:42:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.011+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.011+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.011+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.011+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.012+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.013+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f522880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.022 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.020+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.020+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628907fab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.023 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.021+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.024 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.021+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288fe3e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.025 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.023+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.026 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.023+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628906d3c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.029+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.031 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.029+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289081df80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.034+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.036 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.034+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f522880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.039+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.041 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.039+0000 7f4e3660e640 -1 librbd::ImageState: 0x56288f522a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.042+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.042+0000 7f4e3660e640 -1 librbd::ImageState: 0x5628903f9580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.047+0000 7f4e3660e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.047+0000 7f4e3660e640 -1 librbd::ImageState: 0x56289081cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.049+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.051 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.049+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.057+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.057+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906d3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.060 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628907fab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.062+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.064 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.062+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288fec3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.066+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.069 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.066+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289081df80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.071+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.071+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.073+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.076 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.073+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.084 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.082+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.082+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f3d6200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.085 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.083+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:00.086 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:42:00.083+0000 7f4e35e0d640 -1 librbd::ImageState: 0x5628906a3900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:42:05.780 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:42:05.780 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:05.780 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:05.781 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:06.717 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:42:06.717 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:42:06.718 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863326 2023-10-21T04:42:06.718 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:06.718 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1728514186 ]] 2023-10-21T04:42:06.718 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1728514186 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\7\2\8\5\1\4\1\8\6 ]] 2023-10-21T04:42:06.718 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:42:16.719 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:42:16.719 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:16.719 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:16.722 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:17.051 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.048+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:42:17.051 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.048+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:42:17.059 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:42:17.059 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:42:17.059 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:42:17.060 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:42:17.060 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:42:17.060 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:42:17.060 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:42:17.061 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:17.054+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:42:17.675 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:42:17.676 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:42:17.676 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863337 2023-10-21T04:42:17.677 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:17.677 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1728514186 ]] 2023-10-21T04:42:17.677 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1728514186 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\7\2\8\5\1\4\1\8\6 ]] 2023-10-21T04:42:17.677 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:42:18.624 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:42:18.625 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:42:18.625 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:42:18.625 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:42:18.625 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:42:18.626 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:42:18.626 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:42:18.626 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:42:18.626 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:42:18.626 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:18.621+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:42:27.684 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:42:27.684 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:27.684 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:27.685 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:28.601 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:42:28.601 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:42:28.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863348 2023-10-21T04:42:28.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:28.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1728514186 ]] 2023-10-21T04:42:28.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1728514186 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\7\2\8\5\1\4\1\8\6 ]] 2023-10-21T04:42:28.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:42:29.270 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:42:29.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:42:29.271 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:42:29.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:42:32.334 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:42:32.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:42:32.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:42:32.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:42:32.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:42:32.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:42:32.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:42:32.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:42:32.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:42:32.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:32.332+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:42:34.669 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:42:34.669 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:42:34.670 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:42:34.670 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:42:34.670 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:42:34.670 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:42:34.670 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:42:34.671 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:42:34.671 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:34.667+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:42:38.603 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:42:38.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:38.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:38.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:39.560 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/1728514186 2023-10-21T04:42:39.561 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:42:39.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863359 2023-10-21T04:42:39.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:39.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/1728514186 ]] 2023-10-21T04:42:39.562 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/1728514186 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\7\2\8\5\1\4\1\8\6 ]] 2023-10-21T04:42:39.563 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:42:44.212 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:42:44.209+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:42:44.213 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:42:44.209+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:42:46.361 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:42:46.362 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:42:46.362 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:42:46.362 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:42:46.363 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:42:46.363 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:42:46.363 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:42:46.364 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:42:46.364 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:42:46.364 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:42:46.358+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:42:49.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:42:49.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:49.567 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:49.568 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:50.498 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:42:50.498 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:42:50.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863370 2023-10-21T04:42:50.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:50.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:42:50.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\1\7\2\8\5\1\4\1\8\6 ]] 2023-10-21T04:42:50.499 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3026591092 2023-10-21T04:42:52.110 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3026591092 until 2023-10-21T05:42:51.324675+0000 (3600 sec) 2023-10-21T04:42:52.130 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:42:52.130 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3026591092 2023-10-21T04:42:52.916 INFO:tasks.workunit.client.0.smithi040.stderr:listed 36 entries 2023-10-21T04:42:52.941 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:42:52.941 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:43:00.014 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.012+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289081cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.015 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.012+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f27a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.016 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.015+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.017 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.016+0000 7f4e3560c640 -1 librbd::ImageState: 0x562890372980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.018+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.019 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.018+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289048a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.027 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.026+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.028 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.026+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288f523880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.029+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.030 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.029+0000 7f4e3560c640 -1 librbd::ImageState: 0x56288fab9900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.034+0000 7f4e3560c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.035 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.034+0000 7f4e3560c640 -1 librbd::ImageState: 0x56289072e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.043+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.044 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.043+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f522e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.044+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.045 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.044+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890801280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.047+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.049 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.047+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.052+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.053 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.052+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.058+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.059 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.058+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289072ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.061 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.060+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.062 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.060+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050b600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.069+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.069+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890669480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.070+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.071 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.070+0000 7f4e35e0d640 -1 librbd::ImageState: 0x562890800180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.073+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.074 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.073+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050a780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.079 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.078+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.078+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.089 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.087+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.089 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.087+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.089 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.087+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56289050bd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.090 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.087+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.097 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.095+0000 7f4e35e0d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:00.097 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T04:43:00.095+0000 7f4e35e0d640 -1 librbd::ImageState: 0x56288f78f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:43:02.943 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:43:02.943 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:02.944 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:02.944 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:03.887 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:43:03.888 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:43:03.889 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863383 2023-10-21T04:43:03.890 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:03.890 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:43:03.890 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:43:03.890 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:43:04.022 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:04.021+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:43:04.023 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:04.021+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:43:13.892 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:43:13.892 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:13.892 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:13.892 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:14.827 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:43:14.828 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:43:14.828 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863394 2023-10-21T04:43:14.828 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:14.829 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:43:14.829 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:43:14.829 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:43:24.832 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:24.832 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:43:24.832 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:24.832 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:25.750 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:43:25.751 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:43:25.751 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863405 2023-10-21T04:43:25.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:25.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:43:25.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:43:25.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:43:32.840 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:32.838+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:43:35.753 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:43:35.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:35.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:35.758 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:36.717 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:43:36.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:43:36.719 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863416 2023-10-21T04:43:36.719 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:36.720 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:43:36.720 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:43:36.720 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:43:39.335 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:39.332+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:43:39.335 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:39.332+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:43:39.335 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:39.332+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:43:39.336 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:39.332+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:43:46.724 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:43:46.724 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:46.724 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:46.724 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:47.660 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:43:47.661 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:43:47.662 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863427 2023-10-21T04:43:47.662 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:47.662 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:43:47.662 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:43:47.662 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:43:48.810 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:43:48.807+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:43:57.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:43:57.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:57.666 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:57.667 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:58.607 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3026591092 2023-10-21T04:43:58.607 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:43:58.609 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863438 2023-10-21T04:43:58.609 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:58.609 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3026591092 ]] 2023-10-21T04:43:58.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3026591092 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:43:58.610 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:44:00.175 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:00.169+0000 7f00db7b8640 -1 reset not still connected to 0x55617a505450 2023-10-21T04:44:00.175 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:00.169+0000 7f00db7b8640 -1 reset not still connected to 0x556182c3d6c0 2023-10-21T04:44:00.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:00.169+0000 7f00db7b8640 -1 reset not still connected to 0x5561859bf6c0 2023-10-21T04:44:00.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:00.169+0000 7f00db7b8640 -1 reset not still connected to 0x55618805da00 2023-10-21T04:44:00.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:00.170+0000 7f00db7b8640 -1 reset not still connected to 0x55618a01e5b0 2023-10-21T04:44:01.150 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:01.139+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:44:01.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:01.146+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:44:01.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:01.146+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:44:01.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:01.146+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:44:01.154 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:01.152+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:44:08.610 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:44:08.611 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:08.611 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:08.611 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:09.536 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:44:09.537 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:44:09.538 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863449 2023-10-21T04:44:09.538 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:09.538 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3100140446 ]] 2023-10-21T04:44:09.538 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3100140446 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\0\2\6\5\9\1\0\9\2 ]] 2023-10-21T04:44:09.538 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/3100140446 2023-10-21T04:44:10.834 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:10.832+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:44:11.115 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/3100140446 until 2023-10-21T05:44:10.330391+0000 (3600 sec) 2023-10-21T04:44:11.138 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:44:11.138 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/3100140446 2023-10-21T04:44:11.906 INFO:tasks.workunit.client.0.smithi040.stderr:listed 37 entries 2023-10-21T04:44:11.929 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:44:11.930 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:44:21.935 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:44:21.935 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:21.936 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:21.936 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:22.880 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:44:22.881 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:44:22.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863462 2023-10-21T04:44:22.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:22.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3100140446 ]] 2023-10-21T04:44:22.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3100140446 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\1\0\0\1\4\0\4\4\6 ]] 2023-10-21T04:44:22.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:44:24.915 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:24.910+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:44:28.262 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:44:28.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:44:28.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:44:28.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:44:28.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:44:28.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:44:28.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:44:28.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:44:28.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:44:28.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:28.260+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:44:32.883 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:44:32.885 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:32.885 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:32.885 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:33.820 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:44:33.820 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:44:33.821 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863473 2023-10-21T04:44:33.821 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:33.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3100140446 ]] 2023-10-21T04:44:33.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3100140446 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\1\0\0\1\4\0\4\4\6 ]] 2023-10-21T04:44:33.822 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:44:33.978 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:33.975+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:44:43.823 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:44:43.824 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:43.824 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:43.824 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:44.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:44:44.754 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:44:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863484 2023-10-21T04:44:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3100140446 ]] 2023-10-21T04:44:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3100140446 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\1\0\0\1\4\0\4\4\6 ]] 2023-10-21T04:44:44.756 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:44:45.028 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.025+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:44:45.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.040+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:44:45.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.040+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:44:45.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.040+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:44:45.046 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.043+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:44:45.046 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.043+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:44:45.047 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.043+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:44:45.047 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.043+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:44:45.047 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:45.043+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:44:51.774 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.770+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:44:51.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.770+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:44:51.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.770+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:44:51.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.776+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:44:51.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.776+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:44:51.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.781+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:44:51.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.781+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:44:51.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:44:51.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:44:51.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:44:54.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:44:54.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:54.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:54.759 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:55.704 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:44:55.704 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:44:55.706 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863495 2023-10-21T04:44:55.707 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:55.707 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3100140446 ]] 2023-10-21T04:44:55.707 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3100140446 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\1\0\0\1\4\0\4\4\6 ]] 2023-10-21T04:44:55.707 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:44:56.354 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:56.351+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:44:56.354 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:56.351+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:44:56.354 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:56.351+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:44:56.354 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:44:56.351+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:45:05.710 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:45:05.711 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:05.711 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:05.711 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:06.658 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/3100140446 2023-10-21T04:45:06.658 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:45:06.660 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863506 2023-10-21T04:45:06.660 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:06.660 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/3100140446 ]] 2023-10-21T04:45:06.660 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/3100140446 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\1\0\0\1\4\0\4\4\6 ]] 2023-10-21T04:45:06.660 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:45:07.337 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:45:07.329+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:45:16.678 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:45:16.680 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:16.681 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:16.681 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:17.611 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:45:17.612 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:45:17.613 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863517 2023-10-21T04:45:17.613 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:17.614 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:45:17.614 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\3\1\0\0\1\4\0\4\4\6 ]] 2023-10-21T04:45:17.614 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist add 172.21.15.40:0/2950129325 2023-10-21T04:45:18.821 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:18.818+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:45:18.833 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:18.831+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:45:18.833 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:18.831+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:45:18.833 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:18.831+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:45:18.834 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:18.831+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:45:19.200 INFO:tasks.workunit.client.0.smithi040.stderr:blocklisting 172.21.15.40:0/2950129325 until 2023-10-21T05:45:18.430586+0000 (3600 sec) 2023-10-21T04:45:19.218 INFO:tasks.workunit.client.0.smithi040.stderr:+ ceph osd blocklist ls 2023-10-21T04:45:19.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep -q 172.21.15.40:0/2950129325 2023-10-21T04:45:19.992 INFO:tasks.workunit.client.0.smithi040.stderr:listed 38 entries 2023-10-21T04:45:20.014 INFO:tasks.workunit.client.0.smithi040.stderr:+ PREV_CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:45:20.015 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:45:30.016 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:45:30.017 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:30.017 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:30.026 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:30.919 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:45:30.920 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:45:30.920 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863530 2023-10-21T04:45:30.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:30.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:45:30.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:45:30.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:45:32.150 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:45:32.150 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:45:32.150 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:45:32.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:45:32.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:45:32.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:45:32.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:45:32.151 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:45:32.152 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:45:32.152 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:32.145+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:45:34.041 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:34.039+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:45:34.042 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:34.039+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:45:34.043 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:34.041+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:45:34.043 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:34.041+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:45:38.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.186+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:45:38.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.186+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:45:38.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.192+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:45:38.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.192+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:45:38.196 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.192+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:45:38.198 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.195+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:45:38.198 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.195+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:45:38.198 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:38.195+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:45:40.704 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:45:40.704 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:45:40.704 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:45:40.705 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:45:40.705 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:45:40.705 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:45:40.705 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:45:40.705 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:45:40.706 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:45:40.706 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:45:40.701+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:45:40.922 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:45:40.922 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:40.922 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:40.926 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:41.881 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:45:41.881 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:45:41.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863541 2023-10-21T04:45:41.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:41.882 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:45:41.883 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:45:41.883 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:45:43.548 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:45:43.546+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:45:44.615 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:45:44.612+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:45:44.616 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:45:44.612+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:45:51.884 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:45:51.885 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:51.889 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:51.897 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:52.798 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:45:52.798 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:45:52.799 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863552 2023-10-21T04:45:52.799 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:52.799 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:45:52.800 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:45:52.800 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:45:58.344 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:45:58.343+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:46:02.801 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:46:02.802 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:02.803 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:02.804 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:03.745 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:46:03.746 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:46:03.747 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863563 2023-10-21T04:46:03.747 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:03.747 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:46:03.748 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:46:03.748 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:46:04.026 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.023+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:04.027 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.023+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:46:04.031 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.028+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:46:04.031 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.028+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:46:04.031 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.028+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:46:04.032 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.028+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:46:04.032 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:04.028+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:46:12.153 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:46:12.150+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:46:12.153 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:46:12.150+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:46:12.153 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:46:12.150+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:46:12.154 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:46:12.150+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:46:13.752 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:46:13.752 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:13.752 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:13.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:14.673 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:46:14.674 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:46:14.675 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863574 2023-10-21T04:46:14.675 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:14.675 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:46:14.675 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:46:14.676 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:46:17.513 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.511+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:17.514 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.511+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:46:17.518 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.514+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:46:17.518 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.516+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:46:17.523 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.521+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:46:17.523 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.521+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:46:17.523 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.521+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:46:17.523 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.521+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:46:17.524 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:17.521+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:46:24.297 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:24.295+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:24.297 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:24.295+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:46:24.298 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:24.295+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:46:24.677 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:46:24.677 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:24.677 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:24.678 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:25.412 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:46:25.412 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:46:25.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863585 2023-10-21T04:46:25.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:25.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:46:25.414 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:46:25.415 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:46:26.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:26.271+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:26.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:26.271+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:46:31.977 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:46:31.969+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:46:35.417 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:46:35.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:35.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:35.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:36.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:46:36.340 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:46:36.341 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863596 2023-10-21T04:46:36.342 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:36.342 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:46:36.342 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:46:36.343 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:46:46.343 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:46:46.343 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:46.344 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:46.344 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:47.186 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:47.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:46:47.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:46:47.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:46:47.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:46:47.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:46:47.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:46:47.190 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:46:47.191 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:46:47.191 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.183+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:46:47.315 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:46:47.316 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:46:47.317 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863607 2023-10-21T04:46:47.317 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:47.317 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:46:47.317 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:46:47.318 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:46:47.606 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:47.606 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:46:47.606 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:46:47.606 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:46:47.607 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:46:47.607 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:46:47.607 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:46:47.607 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:46:47.607 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:46:47.608 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:47.600+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:46:49.384 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.378+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:46:49.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.378+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:46:49.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.378+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:46:49.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.378+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:46:49.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.378+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:46:49.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.378+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:46:49.387 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.379+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:46:49.387 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:46:49.387 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:46:49.388 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:46:49.383+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:46:57.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:46:57.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:57.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:57.322 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:58.282 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:46:58.282 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:46:58.283 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863618 2023-10-21T04:46:58.283 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:58.283 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:46:58.283 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:46:58.284 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:47:00.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:00.370+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:47:00.377 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:00.374+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:47:00.383 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:00.379+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:47:00.383 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:00.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:47:00.384 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:00.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:47:08.289 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:08.289 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:08.289 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:08.290 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:47:09.169 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:47:09.169 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:47:09.170 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863629 2023-10-21T04:47:09.170 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:09.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:47:09.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:47:09.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:47:09.557 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:47:09.558 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:47:09.558 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:47:09.558 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:47:09.558 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:47:09.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:47:09.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:47:09.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:47:09.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:47:09.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:09.556+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:47:12.586 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:12.585+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:47:12.587 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:12.585+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:47:12.587 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:12.585+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:47:17.891 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:17.889+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:47:18.578 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:18.576+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:47:19.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:47:19.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:19.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:19.175 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:20.095 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:47:20.096 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:47:20.097 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863640 2023-10-21T04:47:20.098 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:20.098 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:47:20.098 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:47:20.098 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:47:23.349 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:23.348+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:47:23.350 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:23.348+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:47:23.350 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:23.348+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:47:23.350 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:23.348+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:47:27.873 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:47:27.873 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:47:27.873 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:47:27.874 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:47:27.874 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:47:27.874 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:47:27.874 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:47:27.874 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:47:27.875 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:27.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:47:30.100 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:47:30.103 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:30.103 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:30.103 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:30.191 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:30.188+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:47:30.191 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:30.188+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:47:31.035 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:47:31.036 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:47:31.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863651 2023-10-21T04:47:31.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:31.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:47:31.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:47:31.038 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:47:35.679 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:35.677+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:47:35.680 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:35.677+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:47:35.680 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:47:35.677+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:47:36.816 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:36.813+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:47:36.816 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:36.813+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:47:36.816 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:36.813+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:47:41.042 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:47:41.042 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:41.042 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:41.042 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:41.973 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:47:41.974 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:47:41.974 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863661 2023-10-21T04:47:41.974 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:41.974 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:47:41.975 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:47:41.975 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:47:49.349 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.346+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:47:49.350 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.346+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:47:49.350 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.346+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:47:49.355 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:47:49.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:47:49.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:47:49.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:47:49.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:47:49.356 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:47:49.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:47:49.350+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:47:51.976 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:47:51.976 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:51.976 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:51.982 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:52.910 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:47:52.910 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:47:52.911 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863672 2023-10-21T04:47:52.911 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:52.911 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:47:52.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:47:52.912 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:48:02.913 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:48:02.914 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:02.915 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:02.918 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:03.828 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:48:03.829 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:48:03.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863683 2023-10-21T04:48:03.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:03.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:48:03.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:48:03.830 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:48:10.222 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:48:10.222 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:48:10.222 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:48:10.222 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:48:10.223 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:48:10.223 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:48:10.223 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:48:10.223 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:48:10.223 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.215+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:48:10.224 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:10.218+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:48:13.833 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:48:13.833 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:13.833 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:13.834 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:14.768 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:48:14.769 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:48:14.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863694 2023-10-21T04:48:14.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:14.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:48:14.770 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:48:14.771 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:48:18.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:48:18.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:48:18.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:48:18.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:48:18.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:48:18.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:48:18.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:48:18.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:48:18.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:18.650+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:48:24.772 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:48:24.772 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:24.772 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:24.775 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:25.185 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.179+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:48:25.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.188+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:48:25.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.190+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:48:25.202 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.196+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:48:25.202 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.196+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:48:25.202 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.196+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:48:25.203 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:25.196+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:48:25.711 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:48:25.712 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:48:25.713 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863705 2023-10-21T04:48:25.714 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:25.714 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:48:25.714 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:48:25.714 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:48:32.313 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:32.311+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:48:32.313 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:32.311+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:48:33.380 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:33.379+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:48:33.380 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:33.379+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:48:33.381 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:33.379+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:48:33.381 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:33.379+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:48:35.714 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:48:35.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:35.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:35.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:36.667 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:48:36.669 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:48:36.671 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863716 2023-10-21T04:48:36.671 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:36.671 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:48:36.671 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:48:36.672 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:48:41.862 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.859+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:48:41.862 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.859+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:48:41.862 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.860+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:48:41.862 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.860+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:48:41.863 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.860+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:48:41.863 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.860+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:48:41.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.864+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:48:41.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.864+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:48:41.866 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.864+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:48:41.866 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:41.864+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:48:42.222 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:48:42.219+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:48:42.511 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:42.509+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:48:42.516 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:42.515+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:48:42.523 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:42.521+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:48:42.523 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:42.521+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:48:46.673 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:48:46.673 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:46.674 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:46.677 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:47.600 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:48:47.601 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:48:47.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863727 2023-10-21T04:48:47.602 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:47.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:48:47.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:48:47.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:48:52.770 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:48:52.769+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:48:57.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:48:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:57.606 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:58.509 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:48:58.509 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:48:58.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863738 2023-10-21T04:48:58.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:58.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:48:58.511 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:48:58.512 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:49:06.598 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:06.596+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:08.513 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:49:08.513 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:08.514 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:08.520 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:09.454 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.452+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:09.454 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.452+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:49:09.458 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:49:09.460 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.457+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:09.463 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:49:09.465 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863749 2023-10-21T04:49:09.465 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:09.465 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:49:09.466 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:49:09.466 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:49:09.467 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.462+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:49:09.468 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.462+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:49:09.468 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.462+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:09.468 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:09.462+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:49:16.674 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:16.671+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:19.467 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:49:19.468 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:19.469 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:19.469 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:20.431 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:49:20.431 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:49:20.432 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863760 2023-10-21T04:49:20.432 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:20.432 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:49:20.433 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:49:20.433 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:49:20.836 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:49:20.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:20.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:49:20.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:49:20.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:20.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:49:20.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:49:20.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:20.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:49:20.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:20.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:49:25.005 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:49:24.998+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:49:30.434 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:49:30.437 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:30.438 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:30.438 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:31.338 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:49:31.338 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:49:31.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863771 2023-10-21T04:49:31.339 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:31.340 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:49:31.340 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:49:31.340 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:49:33.051 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.038+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:33.052 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.045+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:49:33.053 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.045+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:33.056 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.053+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:49:33.056 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.053+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:49:33.056 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.053+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:33.056 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.053+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:49:33.057 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:33.053+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:49:40.706 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:49:40.707 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:40.707 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:49:40.707 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:49:40.707 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:40.707 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:49:40.708 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:49:40.708 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:40.708 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:40.704+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:49:41.344 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:49:41.344 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:41.344 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:41.345 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:42.292 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:49:42.292 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:49:42.293 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863782 2023-10-21T04:49:42.293 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:42.294 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:49:42.294 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:49:42.294 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:49:43.885 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:49:43.885 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:43.885 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:49:43.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:49:43.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:43.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:49:43.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:49:43.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:43.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:49:43.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:49:43.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:43.882+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:51.941 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:51.932+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:51.941 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:51.932+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:52.295 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:49:52.296 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:52.296 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:52.298 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:52.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:49:52.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:49:52.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:49:52.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:49:52.839 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:49:52.839 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:49:52.839 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:49:52.839 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:49:52.840 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:49:52.834+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:49:53.217 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:49:53.218 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:49:53.218 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863793 2023-10-21T04:49:53.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:53.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:49:53.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:49:53.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:50:03.219 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:50:03.221 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:03.221 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:03.221 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:04.186 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:50:04.186 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:50:04.191 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863804 2023-10-21T04:50:04.191 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:04.192 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:50:04.192 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:50:04.192 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:50:08.259 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:08.256+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:50:14.190 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:50:14.190 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:14.190 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:14.191 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:15.159 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:50:15.159 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:50:15.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863815 2023-10-21T04:50:15.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:15.160 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:50:15.161 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:50:15.161 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:50:18.593 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:18.589+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:50:23.153 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:23.149+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:50:25.162 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:50:25.162 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:25.166 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:25.166 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:26.107 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:50:26.107 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:50:26.109 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863826 2023-10-21T04:50:26.109 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:26.109 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:50:26.109 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:50:26.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:50:29.389 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:29.387+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:50:29.390 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:29.387+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:50:29.390 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:29.387+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:50:29.390 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:50:29.387+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:50:36.111 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:36.112 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:50:36.114 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:36.114 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:37.046 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:50:37.048 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:50:37.050 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863837 2023-10-21T04:50:37.050 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:37.050 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:50:37.050 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:50:37.050 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:50:38.402 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.398+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:50:38.402 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.398+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:50:38.404 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.401+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:50:38.406 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.404+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:50:38.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.411+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:50:38.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.411+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:50:38.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.411+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:50:38.419 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.411+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:50:38.419 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.411+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:50:38.792 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.788+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:50:38.792 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.788+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:50:38.792 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.788+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:50:38.792 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:38.788+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:50:41.595 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:41.592+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:50:41.603 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:41.599+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:50:47.052 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:50:47.052 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:47.052 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:47.058 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:48.002 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:50:48.003 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:50:48.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863848 2023-10-21T04:50:48.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:48.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:50:48.004 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:50:48.005 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:50:57.568 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:50:57.568 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:50:57.569 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:50:57.569 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:50:57.569 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:50:57.569 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:50:57.569 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:50:57.570 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:50:57.570 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:50:57.570 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:50:57.564+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:50:58.006 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:50:58.006 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:58.006 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:58.009 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:58.926 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:50:58.926 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:50:58.927 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863858 2023-10-21T04:50:58.927 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:58.927 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:50:58.927 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:50:58.927 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:51:05.468 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:51:05.469 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:51:05.469 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:51:05.469 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:51:05.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:05.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:51:05.470 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:51:05.471 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:51:05.471 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:05.466+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:51:07.491 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:07.489+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:51:07.492 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:07.489+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:51:08.932 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:51:08.932 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:08.932 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:08.933 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:09.897 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:51:09.897 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:51:09.898 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863869 2023-10-21T04:51:09.898 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:09.899 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:51:09.899 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:51:09.899 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:51:19.905 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:51:19.905 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:19.906 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:19.906 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:20.849 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:51:20.849 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:51:20.849 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863880 2023-10-21T04:51:20.850 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:20.850 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:51:20.851 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:51:20.851 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:51:25.514 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:25.512+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:51:25.514 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:25.512+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:51:25.514 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:25.512+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:25.518 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:25.517+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:25.542 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:25.540+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:51:25.542 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:25.540+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:51:30.851 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:51:30.854 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:30.854 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:30.854 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:31.459 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:31.450+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:51:31.461 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:31.458+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:31.823 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:51:31.823 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:51:31.824 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863891 2023-10-21T04:51:31.824 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:31.824 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:51:31.824 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:51:31.825 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:51:37.555 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.551+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:51:37.555 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.551+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:51:37.560 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.557+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:51:37.560 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.557+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:51:37.560 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.557+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:37.561 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.559+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:51:37.561 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.559+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:51:37.561 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.559+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:51:37.561 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.559+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:51:37.562 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:37.559+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:51:41.828 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:51:41.829 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:41.829 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:41.829 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:42.781 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:51:42.781 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:51:42.782 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863902 2023-10-21T04:51:42.782 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:42.782 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:51:42.782 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:51:42.783 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:51:44.887 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:51:44.884+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:51:44.887 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:51:44.884+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:51:44.887 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:51:44.884+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:51:46.717 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:51:46.718 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:51:46.719 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:51:46.719 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:51:46.719 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:46.719 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:51:46.719 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:51:46.720 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:51:46.720 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:51:46.720 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:46.715+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:51:50.253 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.250+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:51:50.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.258+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:51:50.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.258+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:51:50.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.261+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:51:50.264 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.261+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:51:50.270 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.266+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:51:50.271 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.266+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:51:50.271 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.266+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:51:50.272 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.266+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:51:50.272 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:51:50.266+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:51:52.785 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:51:52.785 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:52.785 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:52.790 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:53.756 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:51:53.756 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:51:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863913 2023-10-21T04:51:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:51:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:51:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:51:58.583 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:51:58.580+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:52:03.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:52:03.760 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:03.760 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:03.760 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:04.714 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:52:04.715 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:52:04.716 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863924 2023-10-21T04:52:04.716 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:04.716 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:52:04.716 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:52:04.717 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:52:14.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:52:14.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:14.718 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:14.720 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:15.653 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:52:15.653 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:52:15.654 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863935 2023-10-21T04:52:15.654 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:15.655 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:52:15.655 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:52:15.655 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:52:25.656 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:52:25.657 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:25.657 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:25.660 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:26.596 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:52:26.596 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:52:26.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863946 2023-10-21T04:52:26.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:26.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:52:26.598 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:52:26.598 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:52:32.019 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:52:32.020 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:52:32.020 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:52:32.020 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:52:32.020 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:52:32.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:52:32.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:52:32.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:52:32.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:52:32.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:32.017+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:52:36.537 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:52:36.534+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:52:36.537 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:52:36.534+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:52:36.599 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:52:36.599 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:36.600 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:36.605 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:37.512 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:52:37.512 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:52:37.513 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863957 2023-10-21T04:52:37.514 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:37.514 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:52:37.514 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:52:37.514 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:52:40.730 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:40.728+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:52:40.734 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:40.728+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:52:42.343 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:52:42.335+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:52:42.343 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:52:42.335+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:52:47.519 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:52:47.519 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:47.519 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:47.520 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:48.445 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:52:48.445 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:52:48.446 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863968 2023-10-21T04:52:48.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:48.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:52:48.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:52:48.447 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:52:58.449 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:52:58.449 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:58.450 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:58.455 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:59.388 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:52:59.389 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:52:59.390 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863979 2023-10-21T04:52:59.390 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:59.391 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:52:59.391 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:52:59.391 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:52:59.473 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:52:59.473 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:52:59.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:52:59.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:52:59.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:52:59.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:52:59.474 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:52:59.471+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:53:09.394 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:53:09.394 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:09.395 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:09.395 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:10.313 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:53:10.313 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:53:10.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697863990 2023-10-21T04:53:10.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:10.314 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:53:10.315 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:53:10.315 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:53:20.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:53:20.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:20.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:20.320 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:20.438 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:53:20.434+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:53:21.286 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:53:21.286 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:53:21.287 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864001 2023-10-21T04:53:21.287 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:21.288 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:53:21.288 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:53:21.288 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:53:27.866 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:53:27.862+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:53:31.289 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:53:31.290 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:31.291 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:31.293 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:32.236 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:53:32.236 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:53:32.237 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864012 2023-10-21T04:53:32.238 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:32.238 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:53:32.238 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:53:32.238 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:53:41.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.897+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:53:41.906 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.902+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:53:41.907 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.902+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:53:41.907 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.902+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:53:41.909 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.907+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:53:41.909 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.907+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:53:41.910 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.907+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:53:41.910 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:41.907+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:53:42.239 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:53:42.243 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:42.243 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:42.243 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:43.192 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:53:43.193 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:53:43.193 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864023 2023-10-21T04:53:43.193 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:43.194 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:53:43.194 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:53:43.194 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:53:43.508 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:53:43.505+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:53:45.348 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.342+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:53:45.349 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.342+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:53:45.349 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.342+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:53:45.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.349+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:53:45.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.349+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:53:45.357 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.352+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:53:45.358 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.352+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:53:45.358 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.352+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:53:45.359 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.352+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:53:45.359 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:53:45.352+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:53:48.986 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:53:48.984+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:53:48.997 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:53:48.995+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:53:53.197 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:53:53.198 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:53.198 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:53.198 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:54.128 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:53:54.129 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:53:54.130 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864034 2023-10-21T04:53:54.130 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:54.131 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:53:54.131 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:53:54.131 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:54:04.132 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:54:04.133 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:04.133 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:04.133 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:05.065 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:54:05.066 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:54:05.067 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864045 2023-10-21T04:54:05.067 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:05.067 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:54:05.067 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:54:05.067 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:54:13.653 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.646+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:54:13.653 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.646+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:54:13.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.654+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:54:13.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.654+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:54:13.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.654+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:54:13.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.654+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:54:13.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:54:13.654+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:54:15.068 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:54:15.069 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:15.069 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:15.076 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:15.987 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:54:15.988 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:54:15.989 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864055 2023-10-21T04:54:15.989 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:15.989 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:54:15.990 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:54:15.990 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:54:17.718 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:17.717+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:54:24.524 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:24.523+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:54:24.524 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:24.523+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:54:24.604 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:24.603+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:54:25.991 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:54:25.991 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:25.993 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:25.997 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:26.932 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:54:26.933 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:54:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864066 2023-10-21T04:54:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:54:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:54:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:54:36.936 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:54:36.937 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:36.938 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:36.945 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:37.298 INFO:tasks.workunit.client.0.smithi040.stderr:2023-10-21T04:54:37.296+0000 7f66e6ffd640 0 --2- 172.21.15.40:0/4282958475 >> v2:172.21.15.40:3300/0 conn(0x7f66e00b03a0 0x7f66e00b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T04:54:38.093 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:54:38.093 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:54:38.094 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864078 2023-10-21T04:54:38.094 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:38.094 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:54:38.095 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:54:38.095 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:54:46.966 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:46.964+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:54:46.967 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:46.964+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:54:46.967 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:46.964+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:54:46.967 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:54:46.964+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:54:48.099 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:54:48.099 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:48.099 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:48.100 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:49.034 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:54:49.036 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:54:49.036 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864089 2023-10-21T04:54:49.036 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:49.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:54:49.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:54:49.037 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:54:59.042 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:54:59.042 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:59.043 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:59.043 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:59.996 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:54:59.997 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:54:59.998 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864099 2023-10-21T04:54:59.998 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:59.998 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:54:59.998 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:54:59.999 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:55:07.203 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.200+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:55:07.210 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.208+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:55:07.220 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:55:07.220 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:55:07.220 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:55:07.220 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:55:07.221 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:55:07.221 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:55:07.221 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:07.216+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:55:09.999 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:55:10.000 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:10.000 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:10.002 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:10.922 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:55:10.923 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:55:10.924 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864110 2023-10-21T04:55:10.924 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:10.924 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:55:10.925 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:55:10.925 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:55:20.926 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:20.927 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:20.927 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:55:20.928 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:21.860 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:55:21.861 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:55:21.861 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864121 2023-10-21T04:55:21.862 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:21.862 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:55:21.862 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:55:21.862 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:55:31.864 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:55:31.864 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:31.864 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:31.867 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:32.802 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:55:32.803 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:55:32.804 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864132 2023-10-21T04:55:32.804 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:32.804 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:55:32.804 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:55:32.804 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:55:42.806 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:55:42.806 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:42.807 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:42.817 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:43.660 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:55:43.662 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:55:43.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:55:43.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:55:43.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:55:43.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:55:43.663 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:55:43.664 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:55:43.664 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:55:43.664 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:43.659+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:55:43.753 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:55:43.754 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:55:43.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864143 2023-10-21T04:55:43.754 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:43.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:55:43.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:55:43.755 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:55:44.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.786+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:55:44.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.786+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:55:44.794 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.792+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:55:44.801 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:55:44.801 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:55:44.802 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:55:44.802 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:55:44.802 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:55:44.802 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:55:44.802 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:55:44.797+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:55:48.045 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:55:48.042+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:55:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:55:53.757 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:53.758 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:53.758 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:54.654 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:55:54.654 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:55:54.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864154 2023-10-21T04:55:54.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:54.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:55:54.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:55:54.656 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:56:03.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:56:03.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:03.021 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:56:03.022 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:56:03.022 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:03.022 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:03.022 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:03.022 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:03.023 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:03.023 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:03.016+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:04.658 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:56:04.661 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:04.661 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:04.661 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:04.924 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.920+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:04.924 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.920+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:04.935 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.932+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:04.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.935+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:04.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.935+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:04.938 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.935+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:04.945 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.942+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:04.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:56:04.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:04.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:56:04.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:56:04.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:04.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:04.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:04.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:04.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:04.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:04.965+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:05.603 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:56:05.604 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:56:05.604 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864165 2023-10-21T04:56:05.605 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:05.605 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:56:05.605 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:56:05.605 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:56:06.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:06.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:08.359 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.356+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:08.366 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.364+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:56:08.367 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.364+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:08.375 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.372+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:08.375 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.372+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:08.376 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.372+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:08.376 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.372+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:08.376 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:08.372+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:15.606 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:56:15.607 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:15.607 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:15.612 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:16.526 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:56:16.527 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:56:16.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864176 2023-10-21T04:56:16.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:16.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:56:16.528 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:56:16.529 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:56:21.836 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:56:21.836 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:21.836 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:56:21.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:56:21.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:21.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:21.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:21.837 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:21.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:21.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:21.833+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:26.530 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:56:26.530 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:26.530 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:26.531 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:27.468 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:56:27.468 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:56:27.469 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864187 2023-10-21T04:56:27.469 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:27.469 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:56:27.470 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:56:27.470 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:56:27.890 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:56:27.888+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:56:27.891 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:56:27.888+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:56:29.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:56:29.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:29.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:56:29.098 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:56:29.098 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:29.098 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:29.098 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:29.098 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:29.099 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:29.099 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:29.093+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:37.475 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:56:37.475 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:37.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:37.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:38.413 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:56:38.414 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:56:38.415 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864198 2023-10-21T04:56:38.415 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:38.415 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:56:38.415 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:56:38.415 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:56:48.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:56:48.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:48.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:48.419 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:49.350 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:56:49.350 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:56:49.351 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864209 2023-10-21T04:56:49.351 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:49.351 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:56:49.351 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:56:49.351 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:56:49.873 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:56:49.864+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:56:51.273 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.258+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:56:51.273 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.258+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:56:51.274 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.266+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:56:51.274 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.267+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:56:51.274 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.267+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:56:51.275 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.273+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:56:51.275 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.273+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:56:51.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.273+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:56:51.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.273+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:56:51.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:56:51.273+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:56:59.356 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:56:59.356 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:59.356 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:59.357 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:00.318 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:57:00.319 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:57:00.319 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864220 2023-10-21T04:57:00.320 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:00.320 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:57:00.320 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:57:00.320 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:57:10.324 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:57:10.325 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:10.325 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:10.334 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:11.105 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:57:11.103+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:57:11.105 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:57:11.103+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:57:11.248 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:57:11.248 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:57:11.249 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864231 2023-10-21T04:57:11.250 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:11.250 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:57:11.250 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:57:11.251 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:57:16.536 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:57:16.533+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:57:16.548 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:57:16.546+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:57:16.548 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:57:16.546+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:57:16.549 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:57:16.546+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:57:16.549 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:57:16.546+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:57:16.549 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:57:16.546+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:57:21.251 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:57:21.252 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:21.253 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:21.256 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:22.179 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:57:22.179 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:57:22.180 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864242 2023-10-21T04:57:22.180 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:22.181 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:57:22.181 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:57:22.181 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:57:26.516 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:57:26.512+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:57:32.182 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:57:32.182 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:32.185 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:32.185 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:33.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:57:33.110 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:57:33.112 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864253 2023-10-21T04:57:33.112 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:33.112 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:57:33.112 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:57:33.112 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:57:43.114 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:57:43.114 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:43.114 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:43.118 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:44.055 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:57:44.055 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:57:44.056 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864264 2023-10-21T04:57:44.056 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:44.057 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:57:44.057 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:57:44.057 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:57:45.035 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:57:45.032+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:57:45.035 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:57:45.032+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:57:54.061 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:57:54.061 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:54.061 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:54.062 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:54.979 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:57:54.980 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:57:54.980 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864274 2023-10-21T04:57:54.981 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:54.981 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:57:54.981 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:57:54.981 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:58:04.984 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:58:04.984 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:04.984 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:04.984 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:05.916 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:58:05.919 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:58:05.920 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864285 2023-10-21T04:58:05.920 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:05.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:58:05.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:58:05.921 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:58:06.259 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:58:06.259 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:58:06.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:58:06.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:58:06.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:58:06.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:58:06.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:58:06.261 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:06.248+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:58:13.659 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:13.657+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:58:13.660 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:13.657+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:58:13.660 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:13.657+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:58:13.660 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:13.657+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:58:14.305 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:14.303+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:58:14.306 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:14.303+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:58:14.316 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:14.313+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:58:14.317 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:14.313+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:58:15.922 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:58:15.923 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:15.923 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:15.932 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:16.847 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:58:16.847 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:58:16.848 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864296 2023-10-21T04:58:16.848 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:16.848 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:58:16.849 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:58:16.849 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:58:26.769 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.766+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:58:26.769 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.766+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:58:26.770 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.768+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:58:26.771 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.769+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:58:26.772 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.769+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:58:26.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.777+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:58:26.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.777+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:58:26.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.777+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:58:26.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.777+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:58:26.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:26.777+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:58:26.853 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:58:26.853 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:26.853 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:26.853 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:27.057 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:27.054+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:58:27.057 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:27.054+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:58:27.785 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:58:27.785 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:58:27.786 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864307 2023-10-21T04:58:27.786 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:27.786 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:58:27.786 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:58:27.787 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:58:36.854 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.851+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T04:58:36.854 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.851+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T04:58:36.864 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T04:58:36.864 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T04:58:36.864 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T04:58:36.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:58:36.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:58:36.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:58:36.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:58:36.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:58:36.863+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:58:37.790 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:37.790 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:58:37.791 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:37.793 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:38.729 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:58:38.729 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:58:38.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864318 2023-10-21T04:58:38.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:38.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:58:38.731 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:58:38.731 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:58:48.733 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:58:48.734 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:48.734 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:48.734 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:49.640 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:58:49.640 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:58:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864329 2023-10-21T04:58:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:58:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:58:49.641 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:58:59.020 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:59.017+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:58:59.020 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:59.017+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:58:59.020 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:59.017+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T04:58:59.021 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:58:59.017+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T04:58:59.642 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:58:59.646 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:59.646 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:59.652 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:00.593 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:59:00.593 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:59:00.594 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864340 2023-10-21T04:59:00.594 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:00.595 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:59:00.595 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:59:00.595 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:59:07.499 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:59:07.490+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:59:07.499 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:59:07.490+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:59:09.171 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:59:09.168+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T04:59:09.171 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:59:09.168+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T04:59:09.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:59:09.168+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T04:59:09.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:59:09.169+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T04:59:09.175 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T04:59:09.173+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T04:59:10.600 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:59:10.600 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:10.600 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:10.601 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:11.588 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:59:11.588 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:59:11.589 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864351 2023-10-21T04:59:11.590 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:11.590 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:59:11.590 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:59:11.590 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:59:17.335 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:59:17.332+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T04:59:17.336 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:59:17.332+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:59:21.598 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:59:21.598 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:21.598 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:21.598 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:22.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:59:22.558 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:59:22.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864362 2023-10-21T04:59:22.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:22.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:59:22.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:59:22.559 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:59:32.562 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:32.564 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:32.565 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:59:32.565 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:33.476 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:59:33.476 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:59:33.477 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864373 2023-10-21T04:59:33.477 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:33.478 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:59:33.478 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:59:33.478 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:59:43.479 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:59:43.482 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:43.482 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:43.482 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:44.418 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:59:44.418 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:59:44.419 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864384 2023-10-21T04:59:44.419 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:44.419 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:59:44.420 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:59:44.420 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:59:48.172 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:59:48.167+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T04:59:54.421 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T04:59:54.424 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:54.424 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:54.425 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:55.388 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T04:59:55.388 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T04:59:55.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864395 2023-10-21T04:59:55.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:55.389 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T04:59:55.390 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T04:59:55.390 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T04:59:56.416 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T04:59:56.415+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:00:05.391 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:00:05.393 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:05.393 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:05.393 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:05.660 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:05.658+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:00:05.660 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:05.658+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:00:05.901 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.899+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:00:05.902 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.899+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:00:05.902 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.900+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:00:05.902 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.900+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:00:05.902 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.900+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:00:05.908 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.904+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:00:05.908 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.904+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:00:05.909 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.904+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:00:05.909 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.904+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:00:05.909 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:05.904+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:00:06.333 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:00:06.333 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:00:06.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864406 2023-10-21T05:00:06.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:06.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:00:06.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:00:06.334 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:00:06.766 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:06.762+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:00:11.114 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:11.113+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:00:11.115 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:11.113+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:00:11.115 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:11.113+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:00:11.115 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:11.113+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:00:13.233 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:13.230+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:00:13.233 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:13.230+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:00:16.336 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:00:16.337 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:16.337 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:16.341 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:17.303 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:00:17.304 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:00:17.305 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864417 2023-10-21T05:00:17.305 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:17.305 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:00:17.305 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:00:17.305 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:00:27.307 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:00:27.310 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:27.310 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:27.313 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:28.232 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:00:28.233 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:00:28.233 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864428 2023-10-21T05:00:28.233 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:28.234 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:00:28.234 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:00:28.234 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:00:38.239 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:00:38.239 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:38.240 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:38.240 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:39.217 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:00:39.218 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:00:39.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864439 2023-10-21T05:00:39.219 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:39.220 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:00:39.220 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:00:39.220 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:00:41.976 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.962+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:00:41.980 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:00:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:00:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:00:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.977+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:00:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.977+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:00:41.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.977+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:00:41.982 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:00:41.977+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:00:49.225 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:00:49.225 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:49.225 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:49.226 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:50.167 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:00:50.169 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:00:50.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864450 2023-10-21T05:00:50.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:50.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:00:50.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:00:50.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:00:56.244 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:56.242+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:00:56.245 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:00:56.242+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:00.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:01:00.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:00.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:00.174 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:01.108 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:01:01.108 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:01:01.109 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864461 2023-10-21T05:01:01.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:01.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:01:01.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:01:01.110 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:01:10.138 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.135+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:01:10.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.135+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:01:10.140 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.137+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:01:10.140 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.137+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:01:10.140 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.137+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:01:10.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.139+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:01:10.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.139+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:01:10.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.139+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:01:10.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.139+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:01:10.143 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:10.139+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:01:10.261 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:10.256+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:01:10.262 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:10.256+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:10.262 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:10.256+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:01:10.262 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:10.256+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:01:11.111 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:01:11.112 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:11.115 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:11.115 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:12.027 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:01:12.027 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:01:12.028 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864472 2023-10-21T05:01:12.029 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:12.029 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:01:12.029 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:01:12.029 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:01:15.561 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:15.559+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:20.512 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.507+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:01:20.514 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.512+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:01:20.519 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.514+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:01:20.535 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.532+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:01:20.535 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.532+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:01:20.535 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.532+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:01:20.535 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.532+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:01:20.535 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:20.532+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:01:22.031 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:01:22.031 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:22.031 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:22.031 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:22.941 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:01:22.941 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:01:22.942 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864482 2023-10-21T05:01:22.942 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:22.943 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:01:22.943 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:01:22.943 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:01:26.212 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:26.210+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:01:26.212 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:26.210+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:32.946 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:01:32.946 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:32.948 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:32.948 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:33.868 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:01:33.869 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:01:33.870 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864493 2023-10-21T05:01:33.870 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:33.870 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:01:33.870 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:01:33.871 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:01:34.334 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.331+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:01:34.344 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.342+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:01:34.353 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:01:34.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:01:34.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:01:34.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:01:34.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:01:34.355 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:01:34.355 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:34.348+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:01:39.085 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:39.083+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:43.872 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:01:43.873 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:43.873 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:43.873 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:44.842 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:01:44.843 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:01:44.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864504 2023-10-21T05:01:44.843 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:44.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:01:44.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:01:44.844 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:01:47.569 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:47.565+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:50.086 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:01:50.086 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:01:50.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:01:50.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:01:50.087 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:01:50.088 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:01:50.088 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:01:50.088 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:01:50.088 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:01:50.088 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:50.082+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:01:51.305 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:51.303+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:01:51.306 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:51.303+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:01:51.306 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:51.303+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:01:51.306 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:01:51.303+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:01:54.846 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:01:54.846 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:54.847 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:54.847 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:55.777 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:01:55.778 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:01:55.779 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864515 2023-10-21T05:01:55.779 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:55.780 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ -n 172.21.15.40:0/2950129325 ]] 2023-10-21T05:01:55.780 INFO:tasks.workunit.client.0.smithi040.stderr:+ [[ 172.21.15.40:0/2950129325 != \1\7\2\.\2\1\.\1\5\.\4\0\:\0\/\2\9\5\0\1\2\9\3\2\5 ]] 2023-10-21T05:01:55.780 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:01:56.818 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:01:56.818 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:01:56.818 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:01:56.818 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:01:56.819 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:01:56.819 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:01:56.819 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:01:56.819 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:01:56.819 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:01:56.820 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:01:56.814+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:02:03.039 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:03.034+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:02:05.782 INFO:tasks.workunit.client.0.smithi040.stderr:++ ceph mgr dump 2023-10-21T05:02:05.782 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:05.783 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:05.783 INFO:tasks.workunit.client.0.smithi040.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:06.730 INFO:tasks.workunit.client.0.smithi040.stderr:+ CLIENT_ADDR=172.21.15.40:0/2950129325 2023-10-21T05:02:06.730 INFO:tasks.workunit.client.0.smithi040.stderr:++ date +%s 2023-10-21T05:02:06.731 INFO:tasks.workunit.client.0.smithi040.stderr:+ CURRENT_TIME=1697864526 2023-10-21T05:02:06.731 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:06.732 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i = 1 )) 2023-10-21T05:02:06.732 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:02:06.732 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:02:06.733 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:06.730+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:02:06.734 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:06.730+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:02:06.859 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:02:06.856+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:02:06.860 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:02:06.862 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:02:11.019 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:11.016+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:02:11.019 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:11.016+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:02:11.019 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:11.016+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:02:11.019 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:11.016+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:02:16.863 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:02:16.864 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:02:16.864 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:02:16.928 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:02:16.925+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:02:16.928 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:02:16.932 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:02:23.429 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.420+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:02:23.431 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.429+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:02:23.435 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.432+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:02:23.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.436+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:02:23.438 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.436+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:02:23.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.436+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:02:23.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:23.436+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:02:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:02:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:02:26.934 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:02:27.002 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:02:26.999+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:02:27.002 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:02:27.010 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:02:28.466 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:28.462+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:02:28.466 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:02:28.462+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:02:37.012 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:02:37.012 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:02:37.012 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:02:37.080 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:02:37.078+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:02:37.081 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:02:37.092 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:02:47.094 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:02:47.094 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:02:47.094 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:02:47.163 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:02:47.161+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:02:47.164 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:02:47.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:02:49.074 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:02:49.074 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:02:49.074 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:02:49.075 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:02:49.075 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:02:49.075 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:02:49.075 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:02:49.075 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:02:49.076 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:02:49.076 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:02:49.076 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:02:49.076 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:02:49.076 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:02:49.076 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.070+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:02:49.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.080+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:02:49.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.080+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:02:49.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.080+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:02:49.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:02:49.080+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:02:57.173 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:02:57.173 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:02:57.174 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:02:57.238 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:02:57.237+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:02:57.239 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:02:57.247 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:03:07.248 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:03:07.249 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:03:07.249 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:07.316 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:03:07.314+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:07.317 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:07.322 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:03:07.350 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.344+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:03:07.351 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.349+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:03:07.352 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.350+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:03:07.352 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.350+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:03:07.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.353+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:03:07.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:03:07.354 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:07.353+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:03:17.324 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:03:17.324 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:03:17.325 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:17.393 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:03:17.390+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:17.394 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:17.397 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:03:18.922 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:03:18.920+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:03:27.399 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:03:27.399 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:03:27.400 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:27.464 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:03:27.462+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:27.465 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:27.479 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:03:27.910 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.906+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:03:27.910 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.906+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:03:27.922 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.918+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:03:27.923 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.918+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:03:27.925 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.922+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:03:27.925 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.922+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:03:27.925 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.922+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:03:27.925 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.922+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:03:27.926 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:27.922+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:03:29.384 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:03:29.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:03:29.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:03:29.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:03:29.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:03:29.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:03:29.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:03:29.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:03:29.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:03:29.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:03:29.381+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:03:37.484 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:03:37.484 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:03:37.485 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:37.547 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:03:37.544+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:37.548 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:37.551 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:03:47.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:03:47.557 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:03:47.558 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:47.621 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:03:47.618+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:47.622 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:47.626 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:03:57.628 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:03:57.628 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:03:57.628 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:57.698 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:03:57.696+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:57.699 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:57.703 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:04:00.965 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:04:00.965 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:04:00.965 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:04:00.966 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:04:00.966 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:04:00.966 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:04:00.966 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:04:00.966 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:04:00.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:04:00.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:00.962+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:04:07.705 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:04:07.705 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:04:07.705 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:07.786 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:04:07.782+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:07.786 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:07.792 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:04:17.793 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:04:17.794 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:04:17.794 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:17.868 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:04:17.866+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:17.870 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:17.878 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:04:18.791 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.789+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:04:18.801 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.798+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:04:18.802 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.801+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:04:18.810 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.807+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:04:18.810 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.807+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:04:18.810 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.807+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:04:18.810 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.807+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:04:18.811 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:18.807+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:04:23.747 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:04:23.740+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:04:23.747 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:04:23.740+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:04:27.879 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:04:27.880 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:04:27.880 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:27.946 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:04:27.944+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:27.947 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:27.952 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:04:37.953 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:04:37.953 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:04:37.954 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:38.020 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:04:38.018+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:38.021 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:38.029 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:04:41.950 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.946+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:04:41.957 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.955+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:04:41.964 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.959+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:04:41.964 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.959+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:04:41.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:04:41.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:04:41.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:04:41.980 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:04:41.980 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:41.976+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:04:48.030 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:04:48.031 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:04:48.031 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:48.097 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:04:48.095+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:48.099 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:48.102 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:04:51.511 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:51.506+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:04:51.515 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:51.506+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:04:51.516 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:04:51.508+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:04:58.103 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:04:58.103 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:04:58.104 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:58.167 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:04:58.164+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:58.168 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:58.171 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:05:05.589 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.585+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:05:05.590 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.585+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:05:05.597 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:05:05.597 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:05:05.597 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:05:05.597 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:05:05.598 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:05:05.598 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:05:05.598 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:05:05.598 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:05.594+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:05:07.917 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:07.913+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:05:07.917 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:07.913+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:05:07.917 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:07.914+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:05:08.172 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:05:08.173 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:05:08.173 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:08.236 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:05:08.234+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:08.237 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:08.243 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:05:11.926 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:05:11.926 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:05:11.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:05:11.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:05:11.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:05:11.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:05:11.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:05:11.927 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:11.923+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:05:14.466 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:14.458+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:05:18.244 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:05:18.245 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:05:18.245 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:18.317 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:05:18.315+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:18.318 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:18.331 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:05:27.300 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:27.292+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:05:28.332 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:05:28.332 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:05:28.332 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:28.399 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:05:28.397+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:28.400 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:28.416 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:05:38.419 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:05:38.420 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:05:38.420 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:38.503 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:05:38.502+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:38.504 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:38.515 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:05:40.304 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:40.302+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:05:44.875 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:44.871+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:05:44.876 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:44.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:05:44.876 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:44.871+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:05:44.877 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:44.876+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:05:48.516 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:05:48.517 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:05:48.517 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:48.591 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:05:48.589+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:48.591 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:48.595 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:05:48.653 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.635+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:05:48.654 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.641+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:05:48.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.650+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:05:48.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.653+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:05:48.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.653+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:05:48.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.653+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:05:48.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.653+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:05:48.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:05:48.653+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:05:54.219 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:05:54.215+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:05:54.220 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:05:54.215+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:05:54.220 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:05:54.215+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:05:58.597 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:05:58.598 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:05:58.598 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:58.672 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:05:58.670+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:58.672 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:58.686 INFO:tasks.workunit.client.0.smithi040.stderr:+ sleep 10 2023-10-21T05:06:00.863 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.851+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:06:00.864 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.854+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:06:00.864 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.858+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:06:00.864 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.858+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:06:00.868 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.864+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:06:00.868 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.864+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:06:00.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.864+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:06:00.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.864+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:06:00.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:00.864+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:06:08.689 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i++ )) 2023-10-21T05:06:08.690 INFO:tasks.workunit.client.0.smithi040.stderr:+ (( i <= 24 )) 2023-10-21T05:06:08.690 INFO:tasks.workunit.client.0.smithi040.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-21T05:06:08.690 INFO:tasks.workunit.client.0.smithi040.stderr:+ grep 'every 2m' 2023-10-21T05:06:08.763 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:06:08.761+0000 7f4e31e05640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:08.764 INFO:tasks.workunit.client.0.smithi040.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T05:06:08.921 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-21T05:06:08.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-21T05:06:12.081 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:06:12.158 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c7a987d078454d378ec209776a2fd3cf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:06:12.161 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T05:06:12.211 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-21T05:06:12.220 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T05:06:12.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:06:13.099 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:06:13.122 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2128,"stamp":"2023-10-21T05:06:11.760876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28744804093,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1952464,"num_read_kb":7544734,"num_write":1900955,"num_write_kb":7601595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82682,"ondisk_log_size":82682,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19973580,"kb_used_data":15198324,"kb_used_omap":4167,"kb_used_meta":4771064,"kb_avail":263141940,"statfs":{"total":289910292480,"available":269457346560,"internally_reserved":0,"allocated":15563083776,"data_stored":35164748992,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4267840,"internal_metadata":4885569728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,6,10,10,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19009536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4330,"num_read_kb":17272,"num_write":4615,"num_write_kb":18460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002072"},"pg_stats":[{"pgid":"2.7","version":"134'6357","reported_seq":9496,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337196+0000","last_change":"2023-10-21T03:59:45.537738+0000","last_active":"2023-10-21T05:06:06.337196+0000","last_peered":"2023-10-21T05:06:06.337196+0000","last_clean":"2023-10-21T05:06:06.337196+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:06:06.337196+0000","last_undegraded":"2023-10-21T05:06:06.337196+0000","last_fullsized":"2023-10-21T05:06:06.337196+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:27:35.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'3250","reported_seq":6846,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:05.008490+0000","last_change":"2023-10-21T03:59:45.537129+0000","last_active":"2023-10-21T05:06:05.008490+0000","last_peered":"2023-10-21T05:06:05.008490+0000","last_clean":"2023-10-21T05:06:05.008490+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:06:05.008490+0000","last_undegraded":"2023-10-21T05:06:05.008490+0000","last_fullsized":"2023-10-21T05:06:05.008490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":3250,"log_dups_size":0,"ondisk_log_size":3250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:05:01.589410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'9500","reported_seq":14082,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337247+0000","last_change":"2023-10-21T03:59:45.537287+0000","last_active":"2023-10-21T05:06:06.337247+0000","last_peered":"2023-10-21T05:06:06.337247+0000","last_clean":"2023-10-21T05:06:06.337247+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:06:06.337247+0000","last_undegraded":"2023-10-21T05:06:06.337247+0000","last_fullsized":"2023-10-21T05:06:06.337247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:28:54.781889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'6391","reported_seq":10544,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337036+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T05:06:06.337036+0000","last_peered":"2023-10-21T05:06:06.337036+0000","last_clean":"2023-10-21T05:06:06.337036+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:06:06.337036+0000","last_undegraded":"2023-10-21T05:06:06.337036+0000","last_fullsized":"2023-10-21T05:06:06.337036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:10:11.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'475188","reported_seq":941578,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:07.763225+0000","last_change":"2023-10-21T04:24:04.969084+0000","last_active":"2023-10-21T05:06:07.763225+0000","last_peered":"2023-10-21T05:06:07.763225+0000","last_clean":"2023-10-21T05:06:07.763225+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:07.763225+0000","last_undegraded":"2023-10-21T05:06:07.763225+0000","last_fullsized":"2023-10-21T05:06:07.763225+0000","mapping_epoch":14,"log_start":"133'465154","ondisk_log_start":"133'465154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:58:39.476215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7108915200,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":471194,"num_read_kb":1857896,"num_write":470183,"num_write_kb":1880732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'12866","reported_seq":19128,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.452157+0000","last_change":"2023-10-21T03:59:45.534408+0000","last_active":"2023-10-21T05:06:06.452157+0000","last_peered":"2023-10-21T05:06:06.452157+0000","last_clean":"2023-10-21T05:06:06.452157+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:06:06.452157+0000","last_undegraded":"2023-10-21T05:06:06.452157+0000","last_fullsized":"2023-10-21T05:06:06.452157+0000","mapping_epoch":11,"log_start":"66'2800","ondisk_log_start":"66'2800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:52:44.705411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'480590","reported_seq":953338,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:08.716084+0000","last_change":"2023-10-21T04:24:04.880538+0000","last_active":"2023-10-21T05:06:08.716084+0000","last_peered":"2023-10-21T05:06:08.716084+0000","last_clean":"2023-10-21T05:06:08.716084+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:06:08.716084+0000","last_undegraded":"2023-10-21T05:06:08.716084+0000","last_fullsized":"2023-10-21T05:06:08.716084+0000","mapping_epoch":14,"log_start":"133'470556","ondisk_log_start":"133'470556","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:10:52.146059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7196606464,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":477599,"num_read_kb":1882592,"num_write":475533,"num_write_kb":1902132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"130'157","reported_seq":3260,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.371738+0000","last_change":"2023-10-21T03:59:45.534361+0000","last_active":"2023-10-21T05:06:06.371738+0000","last_peered":"2023-10-21T05:06:06.371738+0000","last_clean":"2023-10-21T05:06:06.371738+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:06:06.371738+0000","last_undegraded":"2023-10-21T05:06:06.371738+0000","last_fullsized":"2023-10-21T05:06:06.371738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:41:49.585980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'502432","reported_seq":993015,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:07.765062+0000","last_change":"2023-10-21T04:24:04.855615+0000","last_active":"2023-10-21T05:06:07.765062+0000","last_peered":"2023-10-21T05:06:07.765062+0000","last_clean":"2023-10-21T05:06:07.765062+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:07.765062+0000","last_undegraded":"2023-10-21T05:06:07.765062+0000","last_fullsized":"2023-10-21T05:06:07.765062+0000","mapping_epoch":14,"log_start":"133'492348","ondisk_log_start":"133'492348","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:58:32.741117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7517605888,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":495671,"num_read_kb":1954800,"num_write":497139,"num_write_kb":1988556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"130'339","reported_seq":6521,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.341540+0000","last_change":"2023-10-21T03:59:45.534393+0000","last_active":"2023-10-21T05:06:06.341540+0000","last_peered":"2023-10-21T05:06:06.341540+0000","last_clean":"2023-10-21T05:06:06.341540+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:06:06.341540+0000","last_undegraded":"2023-10-21T05:06:06.341540+0000","last_fullsized":"2023-10-21T05:06:06.341540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:11:34.716823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'461811","reported_seq":915688,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:08.736835+0000","last_change":"2023-10-21T04:24:04.893357+0000","last_active":"2023-10-21T05:06:08.736835+0000","last_peered":"2023-10-21T05:06:08.736835+0000","last_clean":"2023-10-21T05:06:08.736835+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:06:08.736835+0000","last_undegraded":"2023-10-21T05:06:08.736835+0000","last_fullsized":"2023-10-21T05:06:08.736835+0000","mapping_epoch":14,"log_start":"133'451756","ondisk_log_start":"133'451756","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:12:51.885175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6921080851,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":458541,"num_read_kb":1808376,"num_write":456942,"num_write_kb":1827762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'6318","reported_seq":9388,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337451+0000","last_change":"2023-10-21T03:59:45.536989+0000","last_active":"2023-10-21T05:06:06.337451+0000","last_peered":"2023-10-21T05:06:06.337451+0000","last_clean":"2023-10-21T05:06:06.337451+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:06:06.337451+0000","last_undegraded":"2023-10-21T05:06:06.337451+0000","last_fullsized":"2023-10-21T05:06:06.337451+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:54:11.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":293,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-21T05:04:18.038986+0000","last_change":"2023-10-21T03:59:43.546458+0000","last_active":"2023-10-21T05:04:18.038986+0000","last_peered":"2023-10-21T05:04:18.038986+0000","last_clean":"2023-10-21T05:04:18.038986+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:04:18.038986+0000","last_undegraded":"2023-10-21T05:04:18.038986+0000","last_fullsized":"2023-10-21T05:04:18.038986+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_clean_scrub_stamp":"2023-10-21T03:59:41.541284+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:25:24.228129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28744208403,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1903005,"num_read_kb":7503664,"num_write":1899797,"num_write_kb":7599182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15560278016,"data_stored":35164094502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72006,"internal_metadata":0},"log_size":42378,"ondisk_log_size":42378,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9782636,"kb_used_data":7597028,"kb_used_omap":2059,"kb_used_meta":2183540,"kb_avail":84589204,"statfs":{"total":96636764160,"available":86619344896,"internally_reserved":0,"allocated":7779356672,"data_stored":17580351851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2108495,"internal_metadata":2235945905},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":0.98899999999999999,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.184},"max":{"1min":3.7799999999999998,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.026,"5min":0.92600000000000005,"15min":0.92400000000000004},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.20499999999999999},"max":{"1min":3.2229999999999999,"5min":3.4060000000000001,"15min":7.2709999999999999},"last":0.25}]},{"osd":1,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.381,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.223,"15min":0.182},"max":{"1min":3.859,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.4770000000000001,"15min":1.218},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":6.1130000000000004,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.44500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739169,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4974528,"kb_used_data":3731064,"kb_used_omap":1037,"kb_used_meta":1242418,"kb_avail":89397312,"statfs":{"total":96636764160,"available":91542847488,"internally_reserved":0,"allocated":3820609536,"data_stored":8631771455,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1272236951},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.88900000000000001,"15min":1.0549999999999999},"min":{"1min":0.24399999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.711,"5min":10.711,"15min":37.695999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.048,"15min":1.0820000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20200000000000001},"max":{"1min":10.512,"5min":10.512,"15min":37.731000000000002},"last":1.3600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.1739999999999999,"15min":1.073},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2969999999999997,"5min":5.4199999999999999,"15min":5.9580000000000002},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.577,"5min":1.2589999999999999,"15min":1.117},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.3360000000000003,"5min":5.4580000000000002,"15min":6},"last":1.3400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5216416,"kb_used_data":3870232,"kb_used_omap":1071,"kb_used_meta":1345104,"kb_avail":89155424,"statfs":{"total":96636764160,"available":91295154176,"internally_reserved":0,"allocated":3963117568,"data_stored":8952625686,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1377386872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94099999999999995,"15min":0.80100000000000005},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":3.9460000000000002,"5min":7.7140000000000004,"15min":7.7140000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96199999999999997,"15min":0.85799999999999998},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.9140000000000001,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.23100000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.8319999999999999,"5min":3.7200000000000002,"15min":7.4509999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.0469999999999999,"15min":1.0429999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16976,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3962159104,"data_stored":8952205312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3819429888,"data_stored":8631291923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7778689024,"data_stored":17580597267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:06:13.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:06:13.650 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:13.651 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:06:13.678 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2128,"stamp":"2023-10-21T05:06:11.760876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28744804093,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1952464,"num_read_kb":7544734,"num_write":1900955,"num_write_kb":7601595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82682,"ondisk_log_size":82682,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19973580,"kb_used_data":15198324,"kb_used_omap":4167,"kb_used_meta":4771064,"kb_avail":263141940,"statfs":{"total":289910292480,"available":269457346560,"internally_reserved":0,"allocated":15563083776,"data_stored":35164748992,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4267840,"internal_metadata":4885569728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,6,10,10,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19009536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4330,"num_read_kb":17272,"num_write":4615,"num_write_kb":18460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002072"},"pg_stats":[{"pgid":"2.7","version":"134'6357","reported_seq":9496,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337196+0000","last_change":"2023-10-21T03:59:45.537738+0000","last_active":"2023-10-21T05:06:06.337196+0000","last_peered":"2023-10-21T05:06:06.337196+0000","last_clean":"2023-10-21T05:06:06.337196+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:06:06.337196+0000","last_undegraded":"2023-10-21T05:06:06.337196+0000","last_fullsized":"2023-10-21T05:06:06.337196+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:27:35.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'3250","reported_seq":6846,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:05.008490+0000","last_change":"2023-10-21T03:59:45.537129+0000","last_active":"2023-10-21T05:06:05.008490+0000","last_peered":"2023-10-21T05:06:05.008490+0000","last_clean":"2023-10-21T05:06:05.008490+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:06:05.008490+0000","last_undegraded":"2023-10-21T05:06:05.008490+0000","last_fullsized":"2023-10-21T05:06:05.008490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":3250,"log_dups_size":0,"ondisk_log_size":3250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:05:01.589410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'9500","reported_seq":14082,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337247+0000","last_change":"2023-10-21T03:59:45.537287+0000","last_active":"2023-10-21T05:06:06.337247+0000","last_peered":"2023-10-21T05:06:06.337247+0000","last_clean":"2023-10-21T05:06:06.337247+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:06:06.337247+0000","last_undegraded":"2023-10-21T05:06:06.337247+0000","last_fullsized":"2023-10-21T05:06:06.337247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:28:54.781889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'6391","reported_seq":10544,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337036+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T05:06:06.337036+0000","last_peered":"2023-10-21T05:06:06.337036+0000","last_clean":"2023-10-21T05:06:06.337036+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:06:06.337036+0000","last_undegraded":"2023-10-21T05:06:06.337036+0000","last_fullsized":"2023-10-21T05:06:06.337036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:10:11.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'475188","reported_seq":941578,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:07.763225+0000","last_change":"2023-10-21T04:24:04.969084+0000","last_active":"2023-10-21T05:06:07.763225+0000","last_peered":"2023-10-21T05:06:07.763225+0000","last_clean":"2023-10-21T05:06:07.763225+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:07.763225+0000","last_undegraded":"2023-10-21T05:06:07.763225+0000","last_fullsized":"2023-10-21T05:06:07.763225+0000","mapping_epoch":14,"log_start":"133'465154","ondisk_log_start":"133'465154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:58:39.476215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7108915200,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":471194,"num_read_kb":1857896,"num_write":470183,"num_write_kb":1880732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'12866","reported_seq":19128,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.452157+0000","last_change":"2023-10-21T03:59:45.534408+0000","last_active":"2023-10-21T05:06:06.452157+0000","last_peered":"2023-10-21T05:06:06.452157+0000","last_clean":"2023-10-21T05:06:06.452157+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:06:06.452157+0000","last_undegraded":"2023-10-21T05:06:06.452157+0000","last_fullsized":"2023-10-21T05:06:06.452157+0000","mapping_epoch":11,"log_start":"66'2800","ondisk_log_start":"66'2800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:52:44.705411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'480590","reported_seq":953338,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:08.716084+0000","last_change":"2023-10-21T04:24:04.880538+0000","last_active":"2023-10-21T05:06:08.716084+0000","last_peered":"2023-10-21T05:06:08.716084+0000","last_clean":"2023-10-21T05:06:08.716084+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:06:08.716084+0000","last_undegraded":"2023-10-21T05:06:08.716084+0000","last_fullsized":"2023-10-21T05:06:08.716084+0000","mapping_epoch":14,"log_start":"133'470556","ondisk_log_start":"133'470556","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:10:52.146059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7196606464,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":477599,"num_read_kb":1882592,"num_write":475533,"num_write_kb":1902132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"130'157","reported_seq":3260,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.371738+0000","last_change":"2023-10-21T03:59:45.534361+0000","last_active":"2023-10-21T05:06:06.371738+0000","last_peered":"2023-10-21T05:06:06.371738+0000","last_clean":"2023-10-21T05:06:06.371738+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:06:06.371738+0000","last_undegraded":"2023-10-21T05:06:06.371738+0000","last_fullsized":"2023-10-21T05:06:06.371738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:41:49.585980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'502432","reported_seq":993015,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:07.765062+0000","last_change":"2023-10-21T04:24:04.855615+0000","last_active":"2023-10-21T05:06:07.765062+0000","last_peered":"2023-10-21T05:06:07.765062+0000","last_clean":"2023-10-21T05:06:07.765062+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:07.765062+0000","last_undegraded":"2023-10-21T05:06:07.765062+0000","last_fullsized":"2023-10-21T05:06:07.765062+0000","mapping_epoch":14,"log_start":"133'492348","ondisk_log_start":"133'492348","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:58:32.741117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7517605888,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":495671,"num_read_kb":1954800,"num_write":497139,"num_write_kb":1988556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"130'339","reported_seq":6521,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.341540+0000","last_change":"2023-10-21T03:59:45.534393+0000","last_active":"2023-10-21T05:06:06.341540+0000","last_peered":"2023-10-21T05:06:06.341540+0000","last_clean":"2023-10-21T05:06:06.341540+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:06:06.341540+0000","last_undegraded":"2023-10-21T05:06:06.341540+0000","last_fullsized":"2023-10-21T05:06:06.341540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:11:34.716823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'461811","reported_seq":915688,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:08.736835+0000","last_change":"2023-10-21T04:24:04.893357+0000","last_active":"2023-10-21T05:06:08.736835+0000","last_peered":"2023-10-21T05:06:08.736835+0000","last_clean":"2023-10-21T05:06:08.736835+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:06:08.736835+0000","last_undegraded":"2023-10-21T05:06:08.736835+0000","last_fullsized":"2023-10-21T05:06:08.736835+0000","mapping_epoch":14,"log_start":"133'451756","ondisk_log_start":"133'451756","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:12:51.885175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6921080851,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":458541,"num_read_kb":1808376,"num_write":456942,"num_write_kb":1827762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'6318","reported_seq":9388,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337451+0000","last_change":"2023-10-21T03:59:45.536989+0000","last_active":"2023-10-21T05:06:06.337451+0000","last_peered":"2023-10-21T05:06:06.337451+0000","last_clean":"2023-10-21T05:06:06.337451+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:06:06.337451+0000","last_undegraded":"2023-10-21T05:06:06.337451+0000","last_fullsized":"2023-10-21T05:06:06.337451+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:54:11.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":293,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-21T05:04:18.038986+0000","last_change":"2023-10-21T03:59:43.546458+0000","last_active":"2023-10-21T05:04:18.038986+0000","last_peered":"2023-10-21T05:04:18.038986+0000","last_clean":"2023-10-21T05:04:18.038986+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:04:18.038986+0000","last_undegraded":"2023-10-21T05:04:18.038986+0000","last_fullsized":"2023-10-21T05:04:18.038986+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_clean_scrub_stamp":"2023-10-21T03:59:41.541284+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:25:24.228129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28744208403,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1903005,"num_read_kb":7503664,"num_write":1899797,"num_write_kb":7599182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15560278016,"data_stored":35164094502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72006,"internal_metadata":0},"log_size":42378,"ondisk_log_size":42378,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9782636,"kb_used_data":7597028,"kb_used_omap":2059,"kb_used_meta":2183540,"kb_avail":84589204,"statfs":{"total":96636764160,"available":86619344896,"internally_reserved":0,"allocated":7779356672,"data_stored":17580351851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2108495,"internal_metadata":2235945905},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":0.98899999999999999,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.184},"max":{"1min":3.7799999999999998,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.026,"5min":0.92600000000000005,"15min":0.92400000000000004},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.20499999999999999},"max":{"1min":3.2229999999999999,"5min":3.4060000000000001,"15min":7.2709999999999999},"last":0.25}]},{"osd":1,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.381,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.223,"15min":0.182},"max":{"1min":3.859,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.4770000000000001,"15min":1.218},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":6.1130000000000004,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.44500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739169,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4974528,"kb_used_data":3731064,"kb_used_omap":1037,"kb_used_meta":1242418,"kb_avail":89397312,"statfs":{"total":96636764160,"available":91542847488,"internally_reserved":0,"allocated":3820609536,"data_stored":8631771455,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1272236951},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.88900000000000001,"15min":1.0549999999999999},"min":{"1min":0.24399999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.711,"5min":10.711,"15min":37.695999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.048,"15min":1.0820000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20200000000000001},"max":{"1min":10.512,"5min":10.512,"15min":37.731000000000002},"last":1.3600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.1739999999999999,"15min":1.073},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2969999999999997,"5min":5.4199999999999999,"15min":5.9580000000000002},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.577,"5min":1.2589999999999999,"15min":1.117},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.3360000000000003,"5min":5.4580000000000002,"15min":6},"last":1.3400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5216416,"kb_used_data":3870232,"kb_used_omap":1071,"kb_used_meta":1345104,"kb_avail":89155424,"statfs":{"total":96636764160,"available":91295154176,"internally_reserved":0,"allocated":3963117568,"data_stored":8952625686,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1377386872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94099999999999995,"15min":0.80100000000000005},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":3.9460000000000002,"5min":7.7140000000000004,"15min":7.7140000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96199999999999997,"15min":0.85799999999999998},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.9140000000000001,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.23100000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.8319999999999999,"5min":3.7200000000000002,"15min":7.4509999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.0469999999999999,"15min":1.0429999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16976,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3962159104,"data_stored":8952205312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3819429888,"data_stored":8631291923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7778689024,"data_stored":17580597267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:06:13.680 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T05:06:13.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:06:14.211 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:14.212 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:06:14.232 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2129,"stamp":"2023-10-21T05:06:13.761210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28749481725,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1953635,"num_read_kb":7549406,"num_write":1902093,"num_write_kb":7606147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82754,"ondisk_log_size":82754,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19820152,"kb_used_data":15211808,"kb_used_omap":4157,"kb_used_meta":4604162,"kb_avail":263295368,"statfs":{"total":289910292480,"available":269614456832,"internally_reserved":0,"allocated":15576891392,"data_stored":35178556608,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":4714662346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,5,6,15,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23687168,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5501,"num_read_kb":21944,"num_write":5753,"num_write_kb":23012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002069"},"pg_stats":[{"pgid":"2.7","version":"134'6357","reported_seq":9496,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337196+0000","last_change":"2023-10-21T03:59:45.537738+0000","last_active":"2023-10-21T05:06:06.337196+0000","last_peered":"2023-10-21T05:06:06.337196+0000","last_clean":"2023-10-21T05:06:06.337196+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:06:06.337196+0000","last_undegraded":"2023-10-21T05:06:06.337196+0000","last_fullsized":"2023-10-21T05:06:06.337196+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6357,"log_dups_size":0,"ondisk_log_size":6357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:27:35.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'3250","reported_seq":6846,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:05.008490+0000","last_change":"2023-10-21T03:59:45.537129+0000","last_active":"2023-10-21T05:06:05.008490+0000","last_peered":"2023-10-21T05:06:05.008490+0000","last_clean":"2023-10-21T05:06:05.008490+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:06:05.008490+0000","last_undegraded":"2023-10-21T05:06:05.008490+0000","last_fullsized":"2023-10-21T05:06:05.008490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":3250,"log_dups_size":0,"ondisk_log_size":3250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:05:01.589410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'9500","reported_seq":14082,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337247+0000","last_change":"2023-10-21T03:59:45.537287+0000","last_active":"2023-10-21T05:06:06.337247+0000","last_peered":"2023-10-21T05:06:06.337247+0000","last_clean":"2023-10-21T05:06:06.337247+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:06:06.337247+0000","last_undegraded":"2023-10-21T05:06:06.337247+0000","last_fullsized":"2023-10-21T05:06:06.337247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:28:54.781889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'6391","reported_seq":10544,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337036+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T05:06:06.337036+0000","last_peered":"2023-10-21T05:06:06.337036+0000","last_clean":"2023-10-21T05:06:06.337036+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:06:06.337036+0000","last_undegraded":"2023-10-21T05:06:06.337036+0000","last_fullsized":"2023-10-21T05:06:06.337036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:10:11.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'475727","reported_seq":942690,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:12.911386+0000","last_change":"2023-10-21T04:24:04.969084+0000","last_active":"2023-10-21T05:06:12.911386+0000","last_peered":"2023-10-21T05:06:12.911386+0000","last_clean":"2023-10-21T05:06:12.911386+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:12.911386+0000","last_undegraded":"2023-10-21T05:06:12.911386+0000","last_fullsized":"2023-10-21T05:06:12.911386+0000","mapping_epoch":14,"log_start":"133'465654","ondisk_log_start":"133'465654","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:58:39.476215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7111131136,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":471762,"num_read_kb":1860164,"num_write":470722,"num_write_kb":1882888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'12900","reported_seq":19166,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:12.309092+0000","last_change":"2023-10-21T03:59:45.534408+0000","last_active":"2023-10-21T05:06:12.309092+0000","last_peered":"2023-10-21T05:06:12.309092+0000","last_clean":"2023-10-21T05:06:12.309092+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:06:12.309092+0000","last_undegraded":"2023-10-21T05:06:12.309092+0000","last_fullsized":"2023-10-21T05:06:12.309092+0000","mapping_epoch":11,"log_start":"66'2800","ondisk_log_start":"66'2800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:52:44.705411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'480590","reported_seq":953338,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:08.716084+0000","last_change":"2023-10-21T04:24:04.880538+0000","last_active":"2023-10-21T05:06:08.716084+0000","last_peered":"2023-10-21T05:06:08.716084+0000","last_clean":"2023-10-21T05:06:08.716084+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:06:08.716084+0000","last_undegraded":"2023-10-21T05:06:08.716084+0000","last_fullsized":"2023-10-21T05:06:08.716084+0000","mapping_epoch":14,"log_start":"133'470556","ondisk_log_start":"133'470556","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:10:52.146059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7196606464,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":477599,"num_read_kb":1882592,"num_write":475533,"num_write_kb":1902132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"130'157","reported_seq":3262,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:11.458433+0000","last_change":"2023-10-21T03:59:45.534361+0000","last_active":"2023-10-21T05:06:11.458433+0000","last_peered":"2023-10-21T05:06:11.458433+0000","last_clean":"2023-10-21T05:06:11.458433+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:06:11.458433+0000","last_undegraded":"2023-10-21T05:06:11.458433+0000","last_fullsized":"2023-10-21T05:06:11.458433+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:41:49.585980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'503031","reported_seq":994214,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:12.877929+0000","last_change":"2023-10-21T04:24:04.855615+0000","last_active":"2023-10-21T05:06:12.877929+0000","last_peered":"2023-10-21T05:06:12.877929+0000","last_clean":"2023-10-21T05:06:12.877929+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:12.877929+0000","last_undegraded":"2023-10-21T05:06:12.877929+0000","last_fullsized":"2023-10-21T05:06:12.877929+0000","mapping_epoch":14,"log_start":"133'492948","ondisk_log_start":"133'492948","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:58:32.741117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7520067584,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":496274,"num_read_kb":1957204,"num_write":497738,"num_write_kb":1990952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"130'339","reported_seq":6525,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:11.459117+0000","last_change":"2023-10-21T03:59:45.534393+0000","last_active":"2023-10-21T05:06:11.459117+0000","last_peered":"2023-10-21T05:06:11.459117+0000","last_clean":"2023-10-21T05:06:11.459117+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:06:11.459117+0000","last_undegraded":"2023-10-21T05:06:11.459117+0000","last_fullsized":"2023-10-21T05:06:11.459117+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:11:34.716823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'461811","reported_seq":915688,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:08.736835+0000","last_change":"2023-10-21T04:24:04.893357+0000","last_active":"2023-10-21T05:06:08.736835+0000","last_peered":"2023-10-21T05:06:08.736835+0000","last_clean":"2023-10-21T05:06:08.736835+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:06:08.736835+0000","last_undegraded":"2023-10-21T05:06:08.736835+0000","last_fullsized":"2023-10-21T05:06:08.736835+0000","mapping_epoch":14,"log_start":"133'451756","ondisk_log_start":"133'451756","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:12:51.885175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6921080851,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":458541,"num_read_kb":1808376,"num_write":456942,"num_write_kb":1827762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'6318","reported_seq":9388,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:06.337451+0000","last_change":"2023-10-21T03:59:45.536989+0000","last_active":"2023-10-21T05:06:06.337451+0000","last_peered":"2023-10-21T05:06:06.337451+0000","last_clean":"2023-10-21T05:06:06.337451+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:06:06.337451+0000","last_undegraded":"2023-10-21T05:06:06.337451+0000","last_fullsized":"2023-10-21T05:06:06.337451+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:54:11.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":293,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-21T05:04:18.038986+0000","last_change":"2023-10-21T03:59:43.546458+0000","last_active":"2023-10-21T05:04:18.038986+0000","last_peered":"2023-10-21T05:04:18.038986+0000","last_clean":"2023-10-21T05:04:18.038986+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:04:18.038986+0000","last_undegraded":"2023-10-21T05:04:18.038986+0000","last_fullsized":"2023-10-21T05:04:18.038986+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_clean_scrub_stamp":"2023-10-21T03:59:41.541284+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:25:24.228129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28748886035,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1904176,"num_read_kb":7508336,"num_write":1900935,"num_write_kb":7603734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15574372352,"data_stored":35178188838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40245,"ondisk_log_size":40245,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":42412,"ondisk_log_size":42412,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739170,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9620616,"kb_used_data":7606080,"kb_used_omap":2048,"kb_used_meta":2012479,"kb_avail":84751224,"statfs":{"total":96636764160,"available":86785253376,"internally_reserved":0,"allocated":7788625920,"data_stored":17589621099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":2060778683},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":0.98899999999999999,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.184},"max":{"1min":3.7799999999999998,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":1.405},{"interface":"front","average":{"1min":1.026,"5min":0.92600000000000005,"15min":0.92400000000000004},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.20499999999999999},"max":{"1min":3.2229999999999999,"5min":3.4060000000000001,"15min":7.2709999999999999},"last":1.5740000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.381,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.223,"15min":0.182},"max":{"1min":3.859,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.4770000000000001,"15min":1.218},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":6.1130000000000004,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.41199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739169,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4974528,"kb_used_data":3731064,"kb_used_omap":1037,"kb_used_meta":1242418,"kb_avail":89397312,"statfs":{"total":96636764160,"available":91542847488,"internally_reserved":0,"allocated":3820609536,"data_stored":8631771455,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1272236951},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.88900000000000001,"15min":1.0549999999999999},"min":{"1min":0.24399999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.711,"5min":10.711,"15min":37.695999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.048,"15min":1.0820000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20200000000000001},"max":{"1min":10.512,"5min":10.512,"15min":37.731000000000002},"last":1.3600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.1739999999999999,"15min":1.073},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2969999999999997,"5min":5.4199999999999999,"15min":5.9580000000000002},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.577,"5min":1.2589999999999999,"15min":1.117},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.3360000000000003,"5min":5.4580000000000002,"15min":6},"last":1.3400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739170,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5225008,"kb_used_data":3874664,"kb_used_omap":1071,"kb_used_meta":1349264,"kb_avail":89146832,"statfs":{"total":96636764160,"available":91286355968,"internally_reserved":0,"allocated":3967655936,"data_stored":8957164054,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1381646712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,1,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94099999999999995,"15min":0.80100000000000005},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":3.9460000000000002,"5min":7.7140000000000004,"15min":7.7140000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96199999999999997,"15min":0.85799999999999998},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.9140000000000001,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.23100000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.8319999999999999,"5min":3.7200000000000002,"15min":7.4509999999999996},"last":0.5},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.0469999999999999,"15min":1.0429999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3966820352,"data_stored":8956866560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3819429888,"data_stored":8631291923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7788122112,"data_stored":17590030355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:06:14.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T05:06:14.789 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:14.790 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":134,"fsid":"df4211a3-b1f8-4292-becf-1d62cb6e7aab","created":"2023-10-21T03:59:28.552484+0000","modified":"2023-10-21T05:05:14.325995+0000","last_up_change":"2023-10-21T03:59:39.830650+0000","last_in_change":"2023-10-21T03:59:29.510751+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T03:59:41.070351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T03:59:43.852830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-21T03:59:49.718586+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":104,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5f8202a-b1a1-454a-b079-217103b4c22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6801","nonce":3061035859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6803","nonce":3061035859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6807","nonce":3061035859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3061035859},{"type":"v1","addr":"172.21.15.40:6805","nonce":3061035859}]},"public_addr":"172.21.15.40:6801/3061035859","cluster_addr":"172.21.15.40:6803/3061035859","heartbeat_back_addr":"172.21.15.40:6807/3061035859","heartbeat_front_addr":"172.21.15.40:6805/3061035859","state":["exists","up"]},{"osd":1,"uuid":"b3ec9f98-f470-4211-af84-4d4565e48ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6810","nonce":1845782567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6814","nonce":1845782567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6821","nonce":1845782567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1845782567},{"type":"v1","addr":"172.21.15.40:6818","nonce":1845782567}]},"public_addr":"172.21.15.40:6810/1845782567","cluster_addr":"172.21.15.40:6814/1845782567","heartbeat_back_addr":"172.21.15.40:6821/1845782567","heartbeat_front_addr":"172.21.15.40:6818/1845782567","state":["exists","up"]},{"osd":2,"uuid":"b7c30dc1-691d-4c84-a137-16ad328be016","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6811","nonce":3642351925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6815","nonce":3642351925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6823","nonce":3642351925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3642351925},{"type":"v1","addr":"172.21.15.40:6820","nonce":3642351925}]},"public_addr":"172.21.15.40:6811/3642351925","cluster_addr":"172.21.15.40:6815/3642351925","heartbeat_back_addr":"172.21.15.40:6823/3642351925","heartbeat_front_addr":"172.21.15.40:6820/3642351925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.602779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.659903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T03:59:34.753221+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/2950129325":"2023-10-21T05:45:18.150091+0000","172.21.15.40:0/3100140446":"2023-10-21T05:44:10.060047+0000","172.21.15.40:0/3026591092":"2023-10-21T05:42:51.031005+0000","172.21.15.40:0/1728514186":"2023-10-21T05:41:43.087944+0000","172.21.15.40:0/3636778574":"2023-10-21T05:40:34.392087+0000","172.21.15.40:0/2732523112":"2023-10-21T05:39:26.078558+0000","172.21.15.40:0/1528643798":"2023-10-21T05:36:58.128748+0000","172.21.15.40:0/3388017662":"2023-10-21T05:35:49.107158+0000","172.21.15.40:0/2848573237":"2023-10-21T05:11:05.572370+0000","172.21.15.40:0/771202222":"2023-10-21T05:15:50.653666+0000","172.21.15.40:0/3096266244":"2023-10-21T05:25:11.792370+0000","172.21.15.40:0/655843917":"2023-10-21T05:14:41.721114+0000","172.21.15.40:0/796892313":"2023-10-21T05:07:28.672941+0000","172.21.15.40:0/2504071713":"2023-10-21T05:22:53.870624+0000","172.21.15.40:0/3701969523":"2023-10-21T05:06:20.622516+0000","172.21.15.40:0/1587307185":"2023-10-21T05:16:58.912965+0000","172.21.15.40:0/1373339116":"2023-10-21T05:09:57.542159+0000","172.21.15.40:0/2300530352":"2023-10-21T05:13:22.699416+0000","172.21.15.40:0/442953649":"2023-10-21T05:20:36.036798+0000","172.21.15.40:0/2008281116":"2023-10-21T05:08:37.818695+0000","172.21.15.40:0/3960257379":"2023-10-21T05:12:13.775135+0000","172.21.15.40:0/3201339169":"2023-10-21T05:18:08.034586+0000","172.21.15.40:0/3576288111":"2023-10-21T05:26:19.922596+0000","172.21.15.40:0/3705540092":"2023-10-21T05:21:44.954926+0000","172.21.15.40:0/746785310":"2023-10-21T05:24:02.857833+0000","172.21.15.40:0/1663690888":"2023-10-21T05:28:47.910205+0000","172.21.15.40:0/764582085":"2023-10-21T05:27:38.937490+0000","172.21.15.40:0/2568248431":"2023-10-21T05:19:16.918548+0000","172.21.15.40:0/2848450024":"2023-10-21T05:33:32.932485+0000","172.21.15.40:0/3333346210":"2023-10-21T05:34:40.934435+0000","172.21.15.40:0/3788273500":"2023-10-21T05:38:06.967616+0000","172.21.15.40:0/2750481391":"2023-10-21T05:29:56.890764+0000","172.21.15.40:0/2564787531":"2023-10-21T05:31:05.011135+0000","172.21.15.40:0/660687942":"2023-10-21T05:32:24.908237+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T05:06:15.810 INFO:tasks.ceph:Scrubbing osd.0 2023-10-21T05:06:15.811 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:06:16.067 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-21T05:06:16.067 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:06:16.068 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-21T05:06:16.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-21T05:06:16.626 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 0 to deep-scrub 2023-10-21T05:06:16.647 INFO:tasks.ceph:Scrubbing osd.1 2023-10-21T05:06:16.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:06:16.880 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-21T05:06:16.880 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:06:16.881 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-21T05:06:16.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-21T05:06:17.217 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:06:17.214+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:06:17.217 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:06:17.214+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:06:17.217 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:06:17.214+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:06:17.448 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 1 to deep-scrub 2023-10-21T05:06:17.470 INFO:tasks.ceph:Scrubbing osd.2 2023-10-21T05:06:17.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:06:17.688 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-21T05:06:17.689 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:06:17.689 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-21T05:06:17.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-21T05:06:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 2 to deep-scrub 2023-10-21T05:06:18.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:06:18.838 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:18.839 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:06:18.856 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2131,"stamp":"2023-10-21T05:06:17.761871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28753995517,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1954695,"num_read_kb":7553618,"num_write":1903194,"num_write_kb":7610551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82775,"ondisk_log_size":82775,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19900164,"kb_used_data":15225252,"kb_used_omap":4157,"kb_used_meta":4670722,"kb_avail":263215356,"statfs":{"total":289910292480,"available":269532524544,"internally_reserved":0,"allocated":15590658048,"data_stored":35192323264,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":4782819786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,7,8,27,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18710528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4334,"num_read_kb":17256,"num_write":4551,"num_write_kb":18204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001999"},"pg_stats":[{"pgid":"2.7","version":"134'6361","reported_seq":9502,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:13.709987+0000","last_change":"2023-10-21T03:59:45.537738+0000","last_active":"2023-10-21T05:06:13.709987+0000","last_peered":"2023-10-21T05:06:13.709987+0000","last_clean":"2023-10-21T05:06:13.709987+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:06:13.709987+0000","last_undegraded":"2023-10-21T05:06:13.709987+0000","last_fullsized":"2023-10-21T05:06:13.709987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6361,"log_dups_size":0,"ondisk_log_size":6361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:27:35.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'3252","reported_seq":6849,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:11.457013+0000","last_change":"2023-10-21T03:59:45.537129+0000","last_active":"2023-10-21T05:06:11.457013+0000","last_peered":"2023-10-21T05:06:11.457013+0000","last_clean":"2023-10-21T05:06:11.457013+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:06:11.457013+0000","last_undegraded":"2023-10-21T05:06:11.457013+0000","last_fullsized":"2023-10-21T05:06:11.457013+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":3252,"log_dups_size":0,"ondisk_log_size":3252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:05:01.589410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'9506","reported_seq":14091,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:13.722640+0000","last_change":"2023-10-21T03:59:45.537287+0000","last_active":"2023-10-21T05:06:13.722640+0000","last_peered":"2023-10-21T05:06:13.722640+0000","last_clean":"2023-10-21T05:06:13.722640+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:06:13.722640+0000","last_undegraded":"2023-10-21T05:06:13.722640+0000","last_fullsized":"2023-10-21T05:06:13.722640+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:28:54.781889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'6395","reported_seq":10550,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:13.710897+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T05:06:13.710897+0000","last_peered":"2023-10-21T05:06:13.710897+0000","last_clean":"2023-10-21T05:06:13.710897+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:06:13.710897+0000","last_undegraded":"2023-10-21T05:06:13.710897+0000","last_fullsized":"2023-10-21T05:06:13.710897+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6395,"log_dups_size":0,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:10:11.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'475727","reported_seq":942690,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:12.911386+0000","last_change":"2023-10-21T04:24:04.969084+0000","last_active":"2023-10-21T05:06:12.911386+0000","last_peered":"2023-10-21T05:06:12.911386+0000","last_clean":"2023-10-21T05:06:12.911386+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:12.911386+0000","last_undegraded":"2023-10-21T05:06:12.911386+0000","last_fullsized":"2023-10-21T05:06:12.911386+0000","mapping_epoch":14,"log_start":"133'465654","ondisk_log_start":"133'465654","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:58:39.476215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7111131136,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":471762,"num_read_kb":1860164,"num_write":470722,"num_write_kb":1882888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'12900","reported_seq":19166,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:12.309092+0000","last_change":"2023-10-21T03:59:45.534408+0000","last_active":"2023-10-21T05:06:12.309092+0000","last_peered":"2023-10-21T05:06:12.309092+0000","last_clean":"2023-10-21T05:06:12.309092+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:06:12.309092+0000","last_undegraded":"2023-10-21T05:06:12.309092+0000","last_fullsized":"2023-10-21T05:06:12.309092+0000","mapping_epoch":11,"log_start":"66'2800","ondisk_log_start":"66'2800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:52:44.705411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'481144","reported_seq":954454,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:13.765046+0000","last_change":"2023-10-21T04:24:04.880538+0000","last_active":"2023-10-21T05:06:13.765046+0000","last_peered":"2023-10-21T05:06:13.765046+0000","last_clean":"2023-10-21T05:06:13.765046+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:06:13.765046+0000","last_undegraded":"2023-10-21T05:06:13.765046+0000","last_fullsized":"2023-10-21T05:06:13.765046+0000","mapping_epoch":14,"log_start":"133'471056","ondisk_log_start":"133'471056","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:10:52.146059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7198875648,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":478162,"num_read_kb":1884824,"num_write":476087,"num_write_kb":1904348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"130'157","reported_seq":3264,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:16.706213+0000","last_change":"2023-10-21T03:59:45.534361+0000","last_active":"2023-10-21T05:06:16.706213+0000","last_peered":"2023-10-21T05:06:16.706213+0000","last_clean":"2023-10-21T05:06:16.706213+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:06:16.706213+0000","last_undegraded":"2023-10-21T05:06:16.706213+0000","last_fullsized":"2023-10-21T05:06:16.706213+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:41:49.585980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'503031","reported_seq":994214,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:12.877929+0000","last_change":"2023-10-21T04:24:04.855615+0000","last_active":"2023-10-21T05:06:12.877929+0000","last_peered":"2023-10-21T05:06:12.877929+0000","last_clean":"2023-10-21T05:06:12.877929+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:12.877929+0000","last_undegraded":"2023-10-21T05:06:12.877929+0000","last_fullsized":"2023-10-21T05:06:12.877929+0000","mapping_epoch":14,"log_start":"133'492948","ondisk_log_start":"133'492948","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:58:32.741117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7520067584,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":496274,"num_read_kb":1957204,"num_write":497738,"num_write_kb":1990952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"130'339","reported_seq":6529,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:16.579410+0000","last_change":"2023-10-21T03:59:45.534393+0000","last_active":"2023-10-21T05:06:16.579410+0000","last_peered":"2023-10-21T05:06:16.579410+0000","last_clean":"2023-10-21T05:06:16.579410+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:06:16.579410+0000","last_undegraded":"2023-10-21T05:06:16.579410+0000","last_fullsized":"2023-10-21T05:06:16.579410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:11:34.716823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'462358","reported_seq":916730,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:13.797193+0000","last_change":"2023-10-21T04:24:04.893357+0000","last_active":"2023-10-21T05:06:13.797193+0000","last_peered":"2023-10-21T05:06:13.797193+0000","last_clean":"2023-10-21T05:06:13.797193+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:06:13.797193+0000","last_undegraded":"2023-10-21T05:06:13.797193+0000","last_fullsized":"2023-10-21T05:06:13.797193+0000","mapping_epoch":14,"log_start":"133'452356","ondisk_log_start":"133'452356","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:12:51.885175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6923325459,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":459038,"num_read_kb":1810356,"num_write":457489,"num_write_kb":1829950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'6322","reported_seq":9394,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:13.762962+0000","last_change":"2023-10-21T03:59:45.536989+0000","last_active":"2023-10-21T05:06:13.762962+0000","last_peered":"2023-10-21T05:06:13.762962+0000","last_clean":"2023-10-21T05:06:13.762962+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:06:13.762962+0000","last_undegraded":"2023-10-21T05:06:13.762962+0000","last_fullsized":"2023-10-21T05:06:13.762962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:54:11.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":293,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-21T05:04:18.038986+0000","last_change":"2023-10-21T03:59:43.546458+0000","last_active":"2023-10-21T05:04:18.038986+0000","last_peered":"2023-10-21T05:04:18.038986+0000","last_clean":"2023-10-21T05:04:18.038986+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:04:18.038986+0000","last_undegraded":"2023-10-21T05:04:18.038986+0000","last_fullsized":"2023-10-21T05:04:18.038986+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:41.541284+0000","last_clean_scrub_stamp":"2023-10-21T03:59:41.541284+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:25:24.228129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28753399827,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1905236,"num_read_kb":7512548,"num_write":1902036,"num_write_kb":7608138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15587450880,"data_stored":35191267366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40246,"ondisk_log_size":40246,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":42432,"ondisk_log_size":42432,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739171,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9673392,"kb_used_data":7614500,"kb_used_omap":2048,"kb_used_meta":2056831,"kb_avail":84698448,"statfs":{"total":96636764160,"available":86731210752,"internally_reserved":0,"allocated":7797248000,"data_stored":17598243179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":2106195131},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":0.98899999999999999,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.184},"max":{"1min":3.7799999999999998,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":1.359},{"interface":"front","average":{"1min":1.026,"5min":0.92600000000000005,"15min":0.92400000000000004},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.20499999999999999},"max":{"1min":3.2229999999999999,"5min":3.4060000000000001,"15min":7.2709999999999999},"last":3.8159999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.381,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.223,"15min":0.182},"max":{"1min":3.859,"5min":27.196999999999999,"15min":27.196999999999999},"last":2.379},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.4770000000000001,"15min":1.218},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":6.1130000000000004,"5min":27.242999999999999,"15min":27.242999999999999},"last":1.393}]}]},{"osd":1,"up_from":8,"seq":34359739170,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5001764,"kb_used_data":3736088,"kb_used_omap":1037,"kb_used_meta":1264626,"kb_avail":89370076,"statfs":{"total":96636764160,"available":91514957824,"internally_reserved":0,"allocated":3825754112,"data_stored":8636916031,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1294977943},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,3,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.88900000000000001,"15min":1.0549999999999999},"min":{"1min":0.24399999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.711,"5min":10.711,"15min":37.695999999999998},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.048,"15min":1.0820000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20200000000000001},"max":{"1min":10.512,"5min":10.512,"15min":37.731000000000002},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.1739999999999999,"15min":1.073},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2969999999999997,"5min":5.4199999999999999,"15min":5.9580000000000002},"last":0.443},{"interface":"front","average":{"1min":1.577,"5min":1.2589999999999999,"15min":1.117},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.3360000000000003,"5min":5.4580000000000002,"15min":6},"last":4.2400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739170,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5225008,"kb_used_data":3874664,"kb_used_omap":1071,"kb_used_meta":1349264,"kb_avail":89146832,"statfs":{"total":96636764160,"available":91286355968,"internally_reserved":0,"allocated":3967655936,"data_stored":8957164054,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1381646712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,1,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94099999999999995,"15min":0.80100000000000005},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":3.9460000000000002,"5min":7.7140000000000004,"15min":7.7140000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96199999999999997,"15min":0.85799999999999998},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.9140000000000001,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.23100000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.8319999999999999,"5min":3.7200000000000002,"15min":7.4509999999999996},"last":0.5},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.0469999999999999,"15min":1.0429999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3966820352,"data_stored":8956866560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3823939584,"data_stored":8635801619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7796690944,"data_stored":17598599187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:06:18.860 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.861 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.862 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.862 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.863 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.863 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.864 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.865 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.865 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.865 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.866 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.866 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.867 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-21T03:59:41.541284+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=41, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:18.867 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:06:24.060 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:24.058+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:06:24.061 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:24.058+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:06:24.061 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:24.058+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:06:27.309 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.302+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:06:27.310 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.302+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:06:27.311 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.308+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:06:27.317 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.308+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:06:27.317 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.308+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:06:27.318 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.316+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:06:27.318 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.316+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:06:27.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.316+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:06:27.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.316+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:06:27.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:27.316+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:06:32.094 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:06:32.094 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:06:32.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:06:32.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:06:32.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:06:32.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:06:32.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:06:32.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:32.091+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:06:34.180 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.172+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:06:34.180 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.172+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:06:34.181 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:06:34.181 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.179+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:06:34.182 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.179+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:06:34.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.180+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:06:34.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.180+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:06:34.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.180+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:06:34.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.180+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:06:34.194 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:34.190+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:06:37.931 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:06:37.924+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:06:38.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:06:38.990 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:06:38.987+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:06:38.990 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:06:38.987+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:06:39.407 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:06:39.428 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2142,"stamp":"2023-10-21T05:06:37.765531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28787169021,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1962592,"num_read_kb":7585042,"num_write":1911245,"num_write_kb":7642755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23593,"num_omap_keys":314,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82871,"ondisk_log_size":82871,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18856408,"kb_used_data":15289152,"kb_used_omap":4157,"kb_used_meta":3563074,"kb_avail":264259112,"statfs":{"total":289910292480,"available":270601330688,"internally_reserved":0,"allocated":15656091648,"data_stored":35257734856,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":3648588234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,6,6,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16580608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3959,"num_read_kb":15772,"num_write":4013,"num_write_kb":16052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10761,"num_omap_keys":155,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002062"},"pg_stats":[{"pgid":"2.7","version":"135'6407","reported_seq":9566,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.719127+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:06:33.719127+0000","last_peered":"2023-10-21T05:06:33.719127+0000","last_clean":"2023-10-21T05:06:33.719127+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:06:33.719127+0000","last_undegraded":"2023-10-21T05:06:33.719127+0000","last_fullsized":"2023-10-21T05:06:33.719127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'3284","reported_seq":6896,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:32.793593+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:06:32.793593+0000","last_peered":"2023-10-21T05:06:32.793593+0000","last_clean":"2023-10-21T05:06:32.793593+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:06:32.793593+0000","last_undegraded":"2023-10-21T05:06:32.793593+0000","last_fullsized":"2023-10-21T05:06:32.793593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3284,"log_dups_size":0,"ondisk_log_size":3284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'9559","reported_seq":14166,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.725052+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:06:33.725052+0000","last_peered":"2023-10-21T05:06:33.725052+0000","last_clean":"2023-10-21T05:06:33.725052+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:06:33.725052+0000","last_undegraded":"2023-10-21T05:06:33.725052+0000","last_fullsized":"2023-10-21T05:06:33.725052+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'6423","reported_seq":10588,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.714398+0000","last_change":"2023-10-21T03:59:45.537049+0000","last_active":"2023-10-21T05:06:33.714398+0000","last_peered":"2023-10-21T05:06:33.714398+0000","last_clean":"2023-10-21T05:06:33.714398+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:06:33.714398+0000","last_undegraded":"2023-10-21T05:06:33.714398+0000","last_fullsized":"2023-10-21T05:06:33.714398+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:44.525010+0000","last_clean_scrub_stamp":"2023-10-21T03:59:44.525010+0000","objects_scrubbed":0,"log_size":6423,"log_dups_size":0,"ondisk_log_size":6423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'477815","reported_seq":946719,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:06:33.055164+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:06:33.055164+0000","last_peered":"2023-10-21T05:06:33.055164+0000","last_clean":"2023-10-21T05:06:33.055164+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:33.055164+0000","last_undegraded":"2023-10-21T05:06:33.055164+0000","last_fullsized":"2023-10-21T05:06:33.055164+0000","mapping_epoch":14,"log_start":"133'467754","ondisk_log_start":"133'467754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":147,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7119269888,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":473688,"num_read_kb":1867828,"num_write":472689,"num_write_kb":1890756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"135'13016","reported_seq":19308,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:32.709683+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:06:32.709683+0000","last_peered":"2023-10-21T05:06:32.709683+0000","last_clean":"2023-10-21T05:06:32.709683+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:06:32.709683+0000","last_undegraded":"2023-10-21T05:06:32.709683+0000","last_fullsized":"2023-10-21T05:06:32.709683+0000","mapping_epoch":11,"log_start":"68'3000","ondisk_log_start":"68'3000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'483191","reported_seq":958484,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.915372+0000","last_change":"2023-10-21T04:24:04.880538+0000","last_active":"2023-10-21T05:06:33.915372+0000","last_peered":"2023-10-21T05:06:33.915372+0000","last_clean":"2023-10-21T05:06:33.915372+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:06:33.915372+0000","last_undegraded":"2023-10-21T05:06:33.915372+0000","last_fullsized":"2023-10-21T05:06:33.915372+0000","mapping_epoch":14,"log_start":"133'473156","ondisk_log_start":"133'473156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7207301120,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":480140,"num_read_kb":1892708,"num_write":478134,"num_write_kb":1912536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3294,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:31.939385+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:06:31.939385+0000","last_peered":"2023-10-21T05:06:31.939385+0000","last_clean":"2023-10-21T05:06:31.939385+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:06:31.939385+0000","last_undegraded":"2023-10-21T05:06:31.939385+0000","last_fullsized":"2023-10-21T05:06:31.939385+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'505201","reported_seq":998507,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:06:33.037067+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:06:33.037067+0000","last_peered":"2023-10-21T05:06:33.037067+0000","last_clean":"2023-10-21T05:06:33.037067+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:33.037067+0000","last_undegraded":"2023-10-21T05:06:33.037067+0000","last_fullsized":"2023-10-21T05:06:33.037067+0000","mapping_epoch":14,"log_start":"133'495148","ondisk_log_start":"133'495148","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":85,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7528710144,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":498380,"num_read_kb":1965588,"num_write":499837,"num_write_kb":1999348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6565,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:31.937345+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:06:31.937345+0000","last_peered":"2023-10-21T05:06:31.937345+0000","last_clean":"2023-10-21T05:06:31.937345+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:06:31.937345+0000","last_undegraded":"2023-10-21T05:06:31.937345+0000","last_fullsized":"2023-10-21T05:06:31.937345+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'464296","reported_seq":920560,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.942495+0000","last_change":"2023-10-21T04:24:04.893357+0000","last_active":"2023-10-21T05:06:33.942495+0000","last_peered":"2023-10-21T05:06:33.942495+0000","last_clean":"2023-10-21T05:06:33.942495+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:06:33.942495+0000","last_undegraded":"2023-10-21T05:06:33.942495+0000","last_fullsized":"2023-10-21T05:06:33.942495+0000","mapping_epoch":14,"log_start":"133'454256","ondisk_log_start":"133'454256","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6931292179,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":460925,"num_read_kb":1817848,"num_write":459427,"num_write_kb":1837702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'6371","reported_seq":9461,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.721069+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:06:33.721069+0000","last_peered":"2023-10-21T05:06:33.721069+0000","last_clean":"2023-10-21T05:06:33.721069+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:06:33.721069+0000","last_undegraded":"2023-10-21T05:06:33.721069+0000","last_fullsized":"2023-10-21T05:06:33.721069+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":306,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:18.866050+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:06:18.866050+0000","last_peered":"2023-10-21T05:06:18.866050+0000","last_clean":"2023-10-21T05:06:18.866050+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:06:18.866050+0000","last_undegraded":"2023-10-21T05:06:18.866050+0000","last_fullsized":"2023-10-21T05:06:18.866050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28786573331,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1913133,"num_read_kb":7543972,"num_write":1910087,"num_write_kb":7640342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15652843520,"data_stored":35256660006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40189,"ondisk_log_size":40189,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23593,"num_omap_keys":314,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":42583,"ondisk_log_size":42583,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739175,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8427592,"kb_used_data":7646016,"kb_used_omap":2048,"kb_used_meta":779519,"kb_avail":85944248,"statfs":{"total":96636764160,"available":88006909952,"internally_reserved":0,"allocated":7829520384,"data_stored":17630508227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":798227643},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":0.98899999999999999,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.184},"max":{"1min":3.7799999999999998,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.026,"5min":0.92600000000000005,"15min":0.92400000000000004},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.20499999999999999},"max":{"1min":3.2229999999999999,"5min":3.4060000000000001,"15min":7.2709999999999999},"last":0.87}]},{"osd":1,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.381,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.223,"15min":0.182},"max":{"1min":3.859,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.4770000000000001,"15min":1.218},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":6.1130000000000004,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.79400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739174,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5089188,"kb_used_data":3751388,"kb_used_omap":1037,"kb_used_meta":1336754,"kb_avail":89282652,"statfs":{"total":96636764160,"available":91425435648,"internally_reserved":0,"allocated":3841421312,"data_stored":8652575895,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1368837015},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.88900000000000001,"15min":1.0549999999999999},"min":{"1min":0.24399999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.711,"5min":10.711,"15min":37.695999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.048,"15min":1.0820000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20200000000000001},"max":{"1min":10.512,"5min":10.512,"15min":37.731000000000002},"last":1.8160000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:05:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.1739999999999999,"15min":1.073},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2969999999999997,"5min":5.4199999999999999,"15min":5.9580000000000002},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.2589999999999999,"15min":1.117},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.3360000000000003,"5min":5.4580000000000002,"15min":6},"last":1.1399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739174,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5339628,"kb_used_data":3891748,"kb_used_omap":1071,"kb_used_meta":1446800,"kb_avail":89032212,"statfs":{"total":96636764160,"available":91168985088,"internally_reserved":0,"allocated":3985149952,"data_stored":8974650734,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1481523576},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,6,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94099999999999995,"15min":0.80100000000000005},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":3.9460000000000002,"5min":7.7140000000000004,"15min":7.7140000000000004},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96199999999999997,"15min":0.85799999999999998},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.9140000000000001,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.98699999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.23100000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.8319999999999999,"5min":3.7200000000000002,"15min":7.4509999999999996},"last":2.3929999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.0469999999999999,"15min":1.0429999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":2.1739999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3983478784,"data_stored":8973524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3840253952,"data_stored":8652115987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7829110784,"data_stored":17631019027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:06:39.430 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-21T03:59:44.525010+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=44, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:39.430 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:39.430 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:39.431 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:39.431 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:06:39.432 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:06:55.029 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:06:55.026+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:06:59.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:06:59.977 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:06:59.978 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:06:59.998 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2153,"stamp":"2023-10-21T05:06:59.769292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28827526909,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1971953,"num_read_kb":7622370,"num_write":1921061,"num_write_kb":7682019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83167,"ondisk_log_size":83167,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19224344,"kb_used_data":15361088,"kb_used_omap":4157,"kb_used_meta":3859074,"kb_avail":263891176,"statfs":{"total":289910292480,"available":270224564224,"internally_reserved":0,"allocated":15729754112,"data_stored":35331397320,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":3951692234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,7,6,17,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24387584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5621,"num_read_kb":22392,"num_write":5931,"num_write_kb":23724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001995"},"pg_stats":[{"pgid":"2.7","version":"135'6437","reported_seq":9606,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:57.536966+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:06:57.536966+0000","last_peered":"2023-10-21T05:06:57.536966+0000","last_clean":"2023-10-21T05:06:57.536966+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:06:57.536966+0000","last_undegraded":"2023-10-21T05:06:57.536966+0000","last_fullsized":"2023-10-21T05:06:57.536966+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'3300","reported_seq":6917,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:56.999452+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:06:56.999452+0000","last_peered":"2023-10-21T05:06:56.999452+0000","last_clean":"2023-10-21T05:06:56.999452+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:06:56.999452+0000","last_undegraded":"2023-10-21T05:06:56.999452+0000","last_fullsized":"2023-10-21T05:06:56.999452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'9605","reported_seq":14228,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:57.537257+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:06:57.537257+0000","last_peered":"2023-10-21T05:06:57.537257+0000","last_clean":"2023-10-21T05:06:57.537257+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:06:57.537257+0000","last_undegraded":"2023-10-21T05:06:57.537257+0000","last_fullsized":"2023-10-21T05:06:57.537257+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'6477","reported_seq":10661,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:57.537032+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:06:57.537032+0000","last_peered":"2023-10-21T05:06:57.537032+0000","last_clean":"2023-10-21T05:06:57.537032+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:06:57.537032+0000","last_undegraded":"2023-10-21T05:06:57.537032+0000","last_fullsized":"2023-10-21T05:06:57.537032+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'480485","reported_seq":951759,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:06:58.332513+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:06:58.332513+0000","last_peered":"2023-10-21T05:06:58.332513+0000","last_clean":"2023-10-21T05:06:58.332513+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:58.332513+0000","last_undegraded":"2023-10-21T05:06:58.332513+0000","last_fullsized":"2023-10-21T05:06:58.332513+0000","mapping_epoch":14,"log_start":"134'470454","ondisk_log_start":"134'470454","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":376,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 40s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7129452544,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":476045,"num_read_kb":1877236,"num_write":475171,"num_write_kb":1900684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"135'13080","reported_seq":19393,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:57.540212+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:06:57.540212+0000","last_peered":"2023-10-21T05:06:57.540212+0000","last_clean":"2023-10-21T05:06:57.540212+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:06:57.540212+0000","last_undegraded":"2023-10-21T05:06:57.540212+0000","last_fullsized":"2023-10-21T05:06:57.540212+0000","mapping_epoch":11,"log_start":"68'3000","ondisk_log_start":"68'3000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'485824","reported_seq":963534,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:06:59.109067+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:06:59.109067+0000","last_peered":"2023-10-21T05:06:59.109067+0000","last_clean":"2023-10-21T05:06:59.109067+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:06:59.109067+0000","last_undegraded":"2023-10-21T05:06:59.109067+0000","last_fullsized":"2023-10-21T05:06:59.109067+0000","mapping_epoch":14,"log_start":"134'475756","ondisk_log_start":"134'475756","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":248,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 24s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7217307648,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":482538,"num_read_kb":1902260,"num_write":480565,"num_write_kb":1922260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3302,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:52.422882+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:06:52.422882+0000","last_peered":"2023-10-21T05:06:52.422882+0000","last_clean":"2023-10-21T05:06:52.422882+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:06:52.422882+0000","last_undegraded":"2023-10-21T05:06:52.422882+0000","last_fullsized":"2023-10-21T05:06:52.422882+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'507902","reported_seq":1003605,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:06:58.322496+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:06:58.322496+0000","last_peered":"2023-10-21T05:06:58.322496+0000","last_clean":"2023-10-21T05:06:58.322496+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:06:58.322496+0000","last_undegraded":"2023-10-21T05:06:58.322496+0000","last_fullsized":"2023-10-21T05:06:58.322496+0000","mapping_epoch":14,"log_start":"134'497848","ondisk_log_start":"134'497848","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":271,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 37s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7539236864,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":500769,"num_read_kb":1975104,"num_write":502388,"num_write_kb":2009552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6581,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:52.419130+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:06:52.419130+0000","last_peered":"2023-10-21T05:06:52.419130+0000","last_clean":"2023-10-21T05:06:52.419130+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:06:52.419130+0000","last_undegraded":"2023-10-21T05:06:52.419130+0000","last_fullsized":"2023-10-21T05:06:52.419130+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'466648","reported_seq":925138,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:59.114472+0000","last_change":"2023-10-21T05:06:55.388063+0000","last_active":"2023-10-21T05:06:59.114472+0000","last_peered":"2023-10-21T05:06:59.114472+0000","last_clean":"2023-10-21T05:06:59.114472+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:06:59.114472+0000","last_undegraded":"2023-10-21T05:06:59.114472+0000","last_fullsized":"2023-10-21T05:06:59.114472+0000","mapping_epoch":14,"log_start":"134'456556","ondisk_log_start":"134'456556","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6940934163,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":463142,"num_read_kb":1826700,"num_write":461779,"num_write_kb":1847110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'6401","reported_seq":9502,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:57.537446+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:06:57.537446+0000","last_peered":"2023-10-21T05:06:57.537446+0000","last_clean":"2023-10-21T05:06:57.537446+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:06:57.537446+0000","last_undegraded":"2023-10-21T05:06:57.537446+0000","last_fullsized":"2023-10-21T05:06:57.537446+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6401,"log_dups_size":0,"ondisk_log_size":6401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":306,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:18.866050+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:06:18.866050+0000","last_peered":"2023-10-21T05:06:18.866050+0000","last_clean":"2023-10-21T05:06:18.866050+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:06:18.866050+0000","last_undegraded":"2023-10-21T05:06:18.866050+0000","last_fullsized":"2023-10-21T05:06:18.866050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28826931219,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1922494,"num_read_kb":7581300,"num_write":1919903,"num_write_kb":7679606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15725494272,"data_stored":35329310758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40245,"ondisk_log_size":40245,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":42823,"ondisk_log_size":42823,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8590228,"kb_used_data":7678604,"kb_used_omap":2048,"kb_used_meta":909567,"kb_avail":85781612,"statfs":{"total":96636764160,"available":87840370688,"internally_reserved":0,"allocated":7862890496,"data_stored":17663878339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":931396795},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":0.98899999999999999,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.185,"15min":0.184},"max":{"1min":3.7799999999999998,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.435},{"interface":"front","average":{"1min":1.026,"5min":0.92600000000000005,"15min":0.92400000000000004},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.20499999999999999},"max":{"1min":3.2229999999999999,"5min":3.4060000000000001,"15min":7.2709999999999999},"last":0.33900000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.381,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.223,"15min":0.182},"max":{"1min":3.859,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.38},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.4770000000000001,"15min":1.218},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":6.1130000000000004,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.40699999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739179,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5184932,"kb_used_data":3770780,"kb_used_omap":1037,"kb_used_meta":1413106,"kb_avail":89186908,"statfs":{"total":96636764160,"available":91327393792,"internally_reserved":0,"allocated":3861278720,"data_stored":8672433303,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1447021463},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.82699999999999996,"15min":1.0209999999999999},"min":{"1min":0.19500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.0800000000000001,"5min":10.711,"15min":37.695999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.294,"5min":1.1359999999999999,"15min":1.0980000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.4870000000000001,"5min":10.512,"15min":37.731000000000002},"last":0.48499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.1000000000000001,"15min":1.077},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":4.0700000000000003,"5min":4.7409999999999997,"15min":5.9580000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.206,"15min":1.149},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.2400000000000002,"5min":4.7809999999999997,"15min":6},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739179,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5449184,"kb_used_data":3911704,"kb_used_omap":1071,"kb_used_meta":1536400,"kb_avail":88922656,"statfs":{"total":96636764160,"available":91056799744,"internally_reserved":0,"allocated":4005584896,"data_stored":8995085678,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1573273976},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,2,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.90100000000000002,"15min":0.80300000000000005},"min":{"1min":0.255,"5min":0.183,"15min":0.183},"max":{"1min":2.9510000000000001,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":1.1759999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.99299999999999999,"15min":0.85499999999999998},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.0859999999999999,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.23100000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":2.8319999999999999,"5min":3.7200000000000002,"15min":7.4509999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.0469999999999999,"15min":1.0429999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4004081664,"data_stored":8994127872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3859849216,"data_stored":8671711251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7861563392,"data_stored":17663471635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:06:59.999 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:00.000 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:00.000 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:00.001 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:00.001 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:07:01.829 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:01.824+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:07:01.833 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:01.830+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:07:01.833 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:01.831+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:07:01.834 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:01.832+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:07:06.587 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:07:06.584+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:07:20.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:20.551 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:07:20.552 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:07:20.573 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2163,"stamp":"2023-10-21T05:07:19.773119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28859057917,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1979741,"num_read_kb":7653434,"num_write":1928728,"num_write_kb":7712687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83163,"ondisk_log_size":83163,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19455528,"kb_used_data":15421968,"kb_used_omap":4157,"kb_used_meta":4029378,"kb_avail":263659992,"statfs":{"total":289910292480,"available":269987831808,"internally_reserved":0,"allocated":15792095232,"data_stored":35393738440,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":4126083530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,5,12,6,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24166400,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5943,"num_read_kb":23712,"num_write":5882,"num_write_kb":23528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002463"},"pg_stats":[{"pgid":"2.7","version":"135'6452","reported_seq":9629,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:19.084344+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:07:19.084344+0000","last_peered":"2023-10-21T05:07:19.084344+0000","last_clean":"2023-10-21T05:07:19.084344+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:07:19.084344+0000","last_undegraded":"2023-10-21T05:07:19.084344+0000","last_fullsized":"2023-10-21T05:07:19.084344+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'3308","reported_seq":6929,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.870430+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:07:18.870430+0000","last_peered":"2023-10-21T05:07:18.870430+0000","last_clean":"2023-10-21T05:07:18.870430+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:07:18.870430+0000","last_undegraded":"2023-10-21T05:07:18.870430+0000","last_fullsized":"2023-10-21T05:07:18.870430+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'9627","reported_seq":14262,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:19.087038+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:07:19.087038+0000","last_peered":"2023-10-21T05:07:19.087038+0000","last_clean":"2023-10-21T05:07:19.087038+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:07:19.087038+0000","last_undegraded":"2023-10-21T05:07:19.087038+0000","last_fullsized":"2023-10-21T05:07:19.087038+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'6492","reported_seq":10684,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:19.083367+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:07:19.083367+0000","last_peered":"2023-10-21T05:07:19.083367+0000","last_clean":"2023-10-21T05:07:19.083367+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:07:19.083367+0000","last_undegraded":"2023-10-21T05:07:19.083367+0000","last_fullsized":"2023-10-21T05:07:19.083367+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6492,"log_dups_size":0,"ondisk_log_size":6492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'482567","reported_seq":955735,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:18.552609+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:07:18.552609+0000","last_peered":"2023-10-21T05:07:18.552609+0000","last_clean":"2023-10-21T05:07:18.552609+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:07:18.552609+0000","last_undegraded":"2023-10-21T05:07:18.552609+0000","last_fullsized":"2023-10-21T05:07:18.552609+0000","mapping_epoch":14,"log_start":"134'472554","ondisk_log_start":"134'472554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":622,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 60s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7137173504,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":477930,"num_read_kb":1884748,"num_write":477049,"num_write_kb":1908196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"135'13160","reported_seq":19489,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.017050+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:07:18.017050+0000","last_peered":"2023-10-21T05:07:18.017050+0000","last_clean":"2023-10-21T05:07:18.017050+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:07:18.017050+0000","last_undegraded":"2023-10-21T05:07:18.017050+0000","last_fullsized":"2023-10-21T05:07:18.017050+0000","mapping_epoch":11,"log_start":"68'3100","ondisk_log_start":"68'3100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'487833","reported_seq":967550,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:19.255412+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:07:19.255412+0000","last_peered":"2023-10-21T05:07:19.255412+0000","last_clean":"2023-10-21T05:07:19.255412+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:07:19.255412+0000","last_undegraded":"2023-10-21T05:07:19.255412+0000","last_fullsized":"2023-10-21T05:07:19.255412+0000","mapping_epoch":14,"log_start":"134'477757","ondisk_log_start":"134'477757","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":448,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 44s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7224942592,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":484532,"num_read_kb":1910228,"num_write":482412,"num_write_kb":1929648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3310,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:12.901515+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:07:12.901515+0000","last_peered":"2023-10-21T05:07:12.901515+0000","last_clean":"2023-10-21T05:07:12.901515+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:07:12.901515+0000","last_undegraded":"2023-10-21T05:07:12.901515+0000","last_fullsized":"2023-10-21T05:07:12.901515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'510058","reported_seq":1007822,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:18.523909+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:07:18.523909+0000","last_peered":"2023-10-21T05:07:18.523909+0000","last_clean":"2023-10-21T05:07:18.523909+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:07:18.523909+0000","last_undegraded":"2023-10-21T05:07:18.523909+0000","last_fullsized":"2023-10-21T05:07:18.523909+0000","mapping_epoch":14,"log_start":"134'500048","ondisk_log_start":"134'500048","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":407,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7547637760,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":502812,"num_read_kb":1983240,"num_write":504434,"num_write_kb":2017736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6597,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:12.905103+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:07:12.905103+0000","last_peered":"2023-10-21T05:07:12.905103+0000","last_clean":"2023-10-21T05:07:12.905103+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:07:12.905103+0000","last_undegraded":"2023-10-21T05:07:12.905103+0000","last_fullsized":"2023-10-21T05:07:12.905103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'468544","reported_seq":928903,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:19.267643+0000","last_change":"2023-10-21T05:07:18.512439+0000","last_active":"2023-10-21T05:07:19.267643+0000","last_peered":"2023-10-21T05:07:19.267643+0000","last_clean":"2023-10-21T05:07:19.267643+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:07:19.267643+0000","last_undegraded":"2023-10-21T05:07:19.267643+0000","last_fullsized":"2023-10-21T05:07:19.267643+0000","mapping_epoch":14,"log_start":"134'458456","ondisk_log_start":"134'458456","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6948708371,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":465008,"num_read_kb":1834148,"num_write":463675,"num_write_kb":1854694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'6415","reported_seq":9524,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:07:19.096549+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:07:19.096549+0000","last_peered":"2023-10-21T05:07:19.096549+0000","last_clean":"2023-10-21T05:07:19.096549+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:07:19.096549+0000","last_undegraded":"2023-10-21T05:07:19.096549+0000","last_fullsized":"2023-10-21T05:07:19.096549+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":306,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-21T05:06:18.866050+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:06:18.866050+0000","last_peered":"2023-10-21T05:06:18.866050+0000","last_clean":"2023-10-21T05:06:18.866050+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:06:18.866050+0000","last_undegraded":"2023-10-21T05:06:18.866050+0000","last_fullsized":"2023-10-21T05:06:18.866050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28858462227,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1930282,"num_read_kb":7612364,"num_write":1927570,"num_write_kb":7710274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15788027904,"data_stored":35391844390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40187,"ondisk_log_size":40187,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":42877,"ondisk_log_size":42877,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8700844,"kb_used_data":7709092,"kb_used_omap":2048,"kb_used_meta":989695,"kb_avail":85670996,"statfs":{"total":96636764160,"available":87727099904,"internally_reserved":0,"allocated":7894110208,"data_stored":17695098051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1013447867},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.92600000000000005,"15min":0.99299999999999999},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.184},"max":{"1min":1.405,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.91900000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":3.8159999999999998,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.88500000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:06:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":1.357,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":2.379,"5min":27.196999999999999,"15min":27.196999999999999},"last":1.6299999999999999},{"interface":"front","average":{"1min":0.748,"5min":1.476,"15min":1.1899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":2.1440000000000001,"5min":27.242999999999999,"15min":27.242999999999999},"last":1.5900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739183,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5235684,"kb_used_data":3785436,"kb_used_omap":1037,"kb_used_meta":1449202,"kb_avail":89136156,"statfs":{"total":96636764160,"available":91275423744,"internally_reserved":0,"allocated":3876286464,"data_stored":8687441047,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1483983767},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.82699999999999996,"15min":1.0209999999999999},"min":{"1min":0.19500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.0800000000000001,"5min":10.711,"15min":37.695999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.1359999999999999,"15min":1.0980000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.4870000000000001,"5min":10.512,"15min":37.731000000000002},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.1000000000000001,"15min":1.077},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":4.0700000000000003,"5min":4.7409999999999997,"15min":5.9580000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.206,"15min":1.149},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.2400000000000002,"5min":4.7809999999999997,"15min":6},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739183,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5519000,"kb_used_data":3927440,"kb_used_omap":1071,"kb_used_meta":1590480,"kb_avail":88852840,"statfs":{"total":96636764160,"available":90985308160,"internally_reserved":0,"allocated":4021698560,"data_stored":9011199342,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1628651896},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,2,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.90100000000000002,"15min":0.80300000000000005},"min":{"1min":0.255,"5min":0.183,"15min":0.183},"max":{"1min":2.9510000000000001,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":2.1400000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.99299999999999999,"15min":0.85499999999999998},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.0859999999999999,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":2.173}]},{"osd":2,"last update":"Sat Oct 21 05:06:59 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.0349999999999999,"15min":1.0109999999999999},"min":{"1min":0.29599999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":7.4509999999999996},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0649999999999999,"15min":1.052},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.8180000000000001,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4020178944,"data_stored":9010225152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3875176448,"data_stored":8687038483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7892672512,"data_stored":17694580755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:20.575 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:20.576 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:20.576 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:20.576 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:20.577 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:07:22.746 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.743+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:07:22.747 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.743+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:07:22.751 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.749+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:07:22.751 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.749+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:07:22.751 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.749+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:07:22.760 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.757+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:07:22.760 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.757+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:07:22.760 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.757+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:07:22.760 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.757+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:07:22.761 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:22.757+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:07:26.182 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:07:26.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:07:26.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:07:26.183 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:07:26.184 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:07:26.184 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:07:26.184 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:07:26.185 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:07:26.185 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:26.178+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:07:31.674 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.670+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:07:31.679 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.675+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:07:31.679 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.675+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:07:31.679 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.675+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:07:31.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.684+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:07:31.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.684+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:07:31.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.684+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:07:31.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.684+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:07:31.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:07:31.684+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:07:40.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:41.124 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:07:41.125 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:07:41.146 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2174,"stamp":"2023-10-21T05:07:39.776666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28891584253,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1987151,"num_read_kb":7683030,"num_write":1936628,"num_write_kb":7744287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83251,"ondisk_log_size":83251,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19720716,"kb_used_data":15485428,"kb_used_omap":4157,"kb_used_meta":4231106,"kb_avail":263394804,"statfs":{"total":289910292480,"available":269716279296,"internally_reserved":0,"allocated":15857078272,"data_stored":35458699343,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":4332653002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,7,21,22,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20180992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4593,"num_read_kb":18352,"num_write":4898,"num_write_kb":19592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.421015"},"pg_stats":[{"pgid":"2.7","version":"136'6478","reported_seq":9665,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:38.496875+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:07:38.496875+0000","last_peered":"2023-10-21T05:07:38.496875+0000","last_clean":"2023-10-21T05:07:38.496875+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:07:38.496875+0000","last_undegraded":"2023-10-21T05:07:38.496875+0000","last_fullsized":"2023-10-21T05:07:38.496875+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6478,"log_dups_size":0,"ondisk_log_size":6478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'3321","reported_seq":6947,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:36.527879+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:07:36.527879+0000","last_peered":"2023-10-21T05:07:36.527879+0000","last_clean":"2023-10-21T05:07:36.527879+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:07:36.527879+0000","last_undegraded":"2023-10-21T05:07:36.527879+0000","last_fullsized":"2023-10-21T05:07:36.527879+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'9666","reported_seq":14314,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:38.496955+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:07:38.496955+0000","last_peered":"2023-10-21T05:07:38.496955+0000","last_clean":"2023-10-21T05:07:38.496955+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:07:38.496955+0000","last_undegraded":"2023-10-21T05:07:38.496955+0000","last_fullsized":"2023-10-21T05:07:38.496955+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'6518","reported_seq":10719,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:38.496859+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:07:38.496859+0000","last_peered":"2023-10-21T05:07:38.496859+0000","last_clean":"2023-10-21T05:07:38.496859+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:07:38.496859+0000","last_undegraded":"2023-10-21T05:07:38.496859+0000","last_fullsized":"2023-10-21T05:07:38.496859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6518,"log_dups_size":0,"ondisk_log_size":6518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'484612","reported_seq":959673,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:38.669305+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:07:38.669305+0000","last_peered":"2023-10-21T05:07:38.669305+0000","last_clean":"2023-10-21T05:07:38.669305+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:07:38.669305+0000","last_undegraded":"2023-10-21T05:07:38.669305+0000","last_fullsized":"2023-10-21T05:07:38.669305+0000","mapping_epoch":14,"log_start":"134'474554","ondisk_log_start":"134'474554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":800,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 80s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7145017344,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":479801,"num_read_kb":1892224,"num_write":478948,"num_write_kb":1915792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'13208","reported_seq":19554,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:38.497772+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:07:38.497772+0000","last_peered":"2023-10-21T05:07:38.497772+0000","last_clean":"2023-10-21T05:07:38.497772+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:07:38.497772+0000","last_undegraded":"2023-10-21T05:07:38.497772+0000","last_fullsized":"2023-10-21T05:07:38.497772+0000","mapping_epoch":11,"log_start":"70'3200","ondisk_log_start":"70'3200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'489923","reported_seq":971531,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:39.524588+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:07:39.524588+0000","last_peered":"2023-10-21T05:07:39.524588+0000","last_clean":"2023-10-21T05:07:39.524588+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:07:39.524588+0000","last_undegraded":"2023-10-21T05:07:39.524588+0000","last_fullsized":"2023-10-21T05:07:39.524588+0000","mapping_epoch":14,"log_start":"134'479857","ondisk_log_start":"134'479857","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":648,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 64s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7232864256,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":486410,"num_read_kb":1917740,"num_write":484339,"num_write_kb":1937356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3319,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:33.379365+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:07:33.379365+0000","last_peered":"2023-10-21T05:07:33.379365+0000","last_clean":"2023-10-21T05:07:33.379365+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:07:33.379365+0000","last_undegraded":"2023-10-21T05:07:33.379365+0000","last_fullsized":"2023-10-21T05:07:33.379365+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'512271","reported_seq":1011919,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:38.678009+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:07:38.678009+0000","last_peered":"2023-10-21T05:07:38.678009+0000","last_clean":"2023-10-21T05:07:38.678009+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:07:38.678009+0000","last_undegraded":"2023-10-21T05:07:38.678009+0000","last_fullsized":"2023-10-21T05:07:38.678009+0000","mapping_epoch":14,"log_start":"134'502248","ondisk_log_start":"134'502248","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":531,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 77s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7556337664,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":504683,"num_read_kb":1990704,"num_write":506546,"num_write_kb":2026184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6614,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:33.380877+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:07:33.380877+0000","last_peered":"2023-10-21T05:07:33.380877+0000","last_clean":"2023-10-21T05:07:33.380877+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:07:33.380877+0000","last_undegraded":"2023-10-21T05:07:33.380877+0000","last_fullsized":"2023-10-21T05:07:33.380877+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'470506","reported_seq":932658,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:39.534494+0000","last_change":"2023-10-21T05:07:37.431132+0000","last_active":"2023-10-21T05:07:39.534494+0000","last_peered":"2023-10-21T05:07:39.534494+0000","last_clean":"2023-10-21T05:07:39.534494+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:07:39.534494+0000","last_undegraded":"2023-10-21T05:07:39.534494+0000","last_fullsized":"2023-10-21T05:07:39.534494+0000","mapping_epoch":14,"log_start":"134'460456","ondisk_log_start":"134'460456","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6956769299,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":466798,"num_read_kb":1841292,"num_write":465637,"num_write_kb":1862542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'6441","reported_seq":9559,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:38.496982+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:07:38.496982+0000","last_peered":"2023-10-21T05:07:38.496982+0000","last_clean":"2023-10-21T05:07:38.496982+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:07:38.496982+0000","last_undegraded":"2023-10-21T05:07:38.496982+0000","last_fullsized":"2023-10-21T05:07:38.496982+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":307,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:24.347417+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:06:24.347417+0000","last_peered":"2023-10-21T05:06:24.347417+0000","last_clean":"2023-10-21T05:06:24.347417+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:06:24.347417+0000","last_undegraded":"2023-10-21T05:06:24.347417+0000","last_fullsized":"2023-10-21T05:06:24.347417+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28890988563,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1937692,"num_read_kb":7641960,"num_write":1935470,"num_write_kb":7741874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15852474368,"data_stored":35456290854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":42955,"ondisk_log_size":42955,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739187,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8774264,"kb_used_data":7740208,"kb_used_omap":2048,"kb_used_meta":1031999,"kb_avail":85597576,"statfs":{"total":96636764160,"available":87651917824,"internally_reserved":0,"allocated":7925972992,"data_stored":17726953456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1056767163},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.92600000000000005,"15min":0.99299999999999999},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.184},"max":{"1min":1.405,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":3.101},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.91900000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":3.8159999999999998,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:06:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":1.357,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":2.379,"5min":27.196999999999999,"15min":27.196999999999999},"last":3.1389999999999998},{"interface":"front","average":{"1min":0.748,"5min":1.476,"15min":1.1899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":2.1440000000000001,"5min":27.242999999999999,"15min":27.242999999999999},"last":3.0590000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739187,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5331336,"kb_used_data":3801600,"kb_used_omap":1037,"kb_used_meta":1528690,"kb_avail":89040504,"statfs":{"total":96636764160,"available":91177476096,"internally_reserved":0,"allocated":3892838400,"data_stored":8703985604,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1565379479},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.82699999999999996,"15min":1.0209999999999999},"min":{"1min":0.19500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.0800000000000001,"5min":10.711,"15min":37.695999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.1359999999999999,"15min":1.0980000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.4870000000000001,"5min":10.512,"15min":37.731000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:06:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.1000000000000001,"15min":1.077},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":4.0700000000000003,"5min":4.7409999999999997,"15min":5.9580000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.206,"15min":1.149},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.2400000000000002,"5min":4.7809999999999997,"15min":6},"last":0.51300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739187,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5615116,"kb_used_data":3943620,"kb_used_omap":1071,"kb_used_meta":1670416,"kb_avail":88756724,"statfs":{"total":96636764160,"available":90886885376,"internally_reserved":0,"allocated":4038266880,"data_stored":9027760283,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1710506360},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,8,14,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.90100000000000002,"15min":0.80300000000000005},"min":{"1min":0.255,"5min":0.183,"15min":0.183},"max":{"1min":2.9510000000000001,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.99299999999999999,"15min":0.85499999999999998},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.0859999999999999,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.253}]},{"osd":2,"last update":"Sat Oct 21 05:06:59 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.0349999999999999,"15min":1.0109999999999999},"min":{"1min":0.29599999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":7.4509999999999996},"last":0.372},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0649999999999999,"15min":1.052},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.8180000000000001,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":3.1669999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4036583424,"data_stored":9026629632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3891105792,"data_stored":8702967827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7924785152,"data_stored":17726693395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:41.147 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:41.147 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:41.148 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:41.148 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:41.149 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:01.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:08:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:08:01.693 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:08:01.711 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2184,"stamp":"2023-10-21T05:07:59.779915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28919666429,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":1994001,"num_read_kb":7710358,"num_write":1943453,"num_write_kb":7771587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83399,"ondisk_log_size":83399,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20012164,"kb_used_data":15542888,"kb_used_omap":4157,"kb_used_meta":4465090,"kb_avail":263103356,"statfs":{"total":289910292480,"available":269417836544,"internally_reserved":0,"allocated":15915917312,"data_stored":35517538383,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":4572252618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,6,12,23,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19943424,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4917,"num_read_kb":19628,"num_write":4851,"num_write_kb":19404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001936"},"pg_stats":[{"pgid":"2.7","version":"136'6504","reported_seq":9697,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:53.856884+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:07:53.856884+0000","last_peered":"2023-10-21T05:07:53.856884+0000","last_clean":"2023-10-21T05:07:53.856884+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:07:53.856884+0000","last_undegraded":"2023-10-21T05:07:53.856884+0000","last_fullsized":"2023-10-21T05:07:53.856884+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6504,"log_dups_size":0,"ondisk_log_size":6504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'3334","reported_seq":6963,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:51.606758+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:07:51.606758+0000","last_peered":"2023-10-21T05:07:51.606758+0000","last_clean":"2023-10-21T05:07:51.606758+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:07:51.606758+0000","last_undegraded":"2023-10-21T05:07:51.606758+0000","last_fullsized":"2023-10-21T05:07:51.606758+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'9705","reported_seq":14362,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:53.858869+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:07:53.858869+0000","last_peered":"2023-10-21T05:07:53.858869+0000","last_clean":"2023-10-21T05:07:53.858869+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:07:53.858869+0000","last_undegraded":"2023-10-21T05:07:53.858869+0000","last_fullsized":"2023-10-21T05:07:53.858869+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'6544","reported_seq":10751,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:53.856876+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:07:53.856876+0000","last_peered":"2023-10-21T05:07:53.856876+0000","last_clean":"2023-10-21T05:07:53.856876+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:07:53.856876+0000","last_undegraded":"2023-10-21T05:07:53.856876+0000","last_fullsized":"2023-10-21T05:07:53.856876+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'486706","reported_seq":963686,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:58.805977+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:07:58.805977+0000","last_peered":"2023-10-21T05:07:58.805977+0000","last_clean":"2023-10-21T05:07:58.805977+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:07:58.805977+0000","last_undegraded":"2023-10-21T05:07:58.805977+0000","last_fullsized":"2023-10-21T05:07:58.805977+0000","mapping_epoch":14,"log_start":"134'476655","ondisk_log_start":"134'476655","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1004,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 100s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7152955392,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":481706,"num_read_kb":1899836,"num_write":480875,"num_write_kb":1923500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'13272","reported_seq":19630,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:58.238949+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:07:58.238949+0000","last_peered":"2023-10-21T05:07:58.238949+0000","last_clean":"2023-10-21T05:07:58.238949+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:07:58.238949+0000","last_undegraded":"2023-10-21T05:07:58.238949+0000","last_fullsized":"2023-10-21T05:07:58.238949+0000","mapping_epoch":11,"log_start":"70'3200","ondisk_log_start":"70'3200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'491446","reported_seq":974559,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:54.750601+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:07:54.750601+0000","last_peered":"2023-10-21T05:07:54.750601+0000","last_clean":"2023-10-21T05:07:54.750601+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:07:54.750601+0000","last_undegraded":"2023-10-21T05:07:54.750601+0000","last_fullsized":"2023-10-21T05:07:54.750601+0000","mapping_epoch":14,"log_start":"134'481358","ondisk_log_start":"134'481358","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":798,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 79s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7238598656,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":487903,"num_read_kb":1923696,"num_write":485737,"num_write_kb":1942948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3327,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:53.861088+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:07:53.861088+0000","last_peered":"2023-10-21T05:07:53.861088+0000","last_clean":"2023-10-21T05:07:53.861088+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:07:53.861088+0000","last_undegraded":"2023-10-21T05:07:53.861088+0000","last_fullsized":"2023-10-21T05:07:53.861088+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'514452","reported_seq":1016117,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:58.792025+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:07:58.792025+0000","last_peered":"2023-10-21T05:07:58.792025+0000","last_clean":"2023-10-21T05:07:58.792025+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:07:58.792025+0000","last_undegraded":"2023-10-21T05:07:58.792025+0000","last_fullsized":"2023-10-21T05:07:58.792025+0000","mapping_epoch":14,"log_start":"135'504448","ondisk_log_start":"135'504448","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":700,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 97s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7564718080,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":506688,"num_read_kb":1998692,"num_write":508588,"num_write_kb":2034352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6630,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:53.870406+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:07:53.870406+0000","last_peered":"2023-10-21T05:07:53.870406+0000","last_clean":"2023-10-21T05:07:53.870406+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:07:53.870406+0000","last_undegraded":"2023-10-21T05:07:53.870406+0000","last_fullsized":"2023-10-21T05:07:53.870406+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'471964","reported_seq":935570,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:54.772413+0000","last_change":"2023-10-21T05:07:51.616218+0000","last_active":"2023-10-21T05:07:54.772413+0000","last_peered":"2023-10-21T05:07:54.772413+0000","last_clean":"2023-10-21T05:07:54.772413+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:07:54.772413+0000","last_undegraded":"2023-10-21T05:07:54.772413+0000","last_fullsized":"2023-10-21T05:07:54.772413+0000","mapping_epoch":14,"log_start":"134'461956","ondisk_log_start":"134'461956","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6962798611,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":468245,"num_read_kb":1847064,"num_write":467095,"num_write_kb":1868374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'6467","reported_seq":9591,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:53.858411+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:07:53.858411+0000","last_peered":"2023-10-21T05:07:53.858411+0000","last_clean":"2023-10-21T05:07:53.858411+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:07:53.858411+0000","last_undegraded":"2023-10-21T05:07:53.858411+0000","last_fullsized":"2023-10-21T05:07:53.858411+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6467,"log_dups_size":0,"ondisk_log_size":6467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":307,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-21T05:06:24.347417+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:06:24.347417+0000","last_peered":"2023-10-21T05:06:24.347417+0000","last_clean":"2023-10-21T05:06:24.347417+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:06:24.347417+0000","last_undegraded":"2023-10-21T05:06:24.347417+0000","last_fullsized":"2023-10-21T05:06:24.347417+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28919070739,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1944542,"num_read_kb":7669288,"num_write":1942295,"num_write_kb":7769174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15912185856,"data_stored":35516002342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40151,"ondisk_log_size":40151,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":43149,"ondisk_log_size":43149,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739191,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8909304,"kb_used_data":7771120,"kb_used_omap":2048,"kb_used_meta":1136127,"kb_avail":85462536,"statfs":{"total":96636764160,"available":87513636864,"internally_reserved":0,"allocated":7957626880,"data_stored":17758607344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1163394235},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.92600000000000005,"15min":0.99299999999999999},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.184},"max":{"1min":1.405,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.91900000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":3.8159999999999998,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:06:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":1.357,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":2.379,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.748,"5min":1.476,"15min":1.1899999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":2.1440000000000001,"5min":27.242999999999999,"15min":27.242999999999999},"last":1.006}]}]},{"osd":1,"up_from":8,"seq":34359739190,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5400424,"kb_used_data":3813024,"kb_used_omap":1037,"kb_used_meta":1586354,"kb_avail":88971416,"statfs":{"total":96636764160,"available":91106729984,"internally_reserved":0,"allocated":3904536576,"data_stored":8715683780,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1624427415},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:49 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.81999999999999995,"15min":1.0129999999999999},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.8140000000000001,"5min":10.711,"15min":37.695999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.169,"15min":1.085},"min":{"1min":0.22,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":3.056,"5min":10.512,"15min":37.731000000000002},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:07:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.1739999999999999,"15min":1.1120000000000001},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":10.167999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.2170000000000001,"15min":1.1599999999999999},"min":{"1min":0.30299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":6},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739191,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5702436,"kb_used_data":3958744,"kb_used_omap":1071,"kb_used_meta":1742608,"kb_avail":88669404,"statfs":{"total":96636764160,"available":90797469696,"internally_reserved":0,"allocated":4053753856,"data_stored":9043247259,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1784430968},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,7,14,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.81899999999999995},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":3.1949999999999998,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":2.5819999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.997,"15min":0.877},"min":{"1min":0.247,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.2440000000000002,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":2.6160000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:06:59 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.0349999999999999,"15min":1.0109999999999999},"min":{"1min":0.29599999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":7.4509999999999996},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0649999999999999,"15min":1.052},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.8180000000000001,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":3.0249999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4052840448,"data_stored":9042886656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3902808064,"data_stored":8714670099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7956537344,"data_stored":17758445587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:08:01.713 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:01.713 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:01.714 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:01.714 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:01.715 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:12.755 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.753+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:08:12.756 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.753+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:08:12.757 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.754+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:08:12.757 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.754+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:08:12.757 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.754+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:08:12.763 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.760+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:08:12.764 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.760+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:08:12.764 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.760+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:08:12.764 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.760+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:08:12.764 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:12.760+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:08:17.212 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:08:17.204+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:08:21.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:08:22.265 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:08:22.266 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:08:22.285 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2195,"stamp":"2023-10-21T05:08:21.784599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28954973949,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2002443,"num_read_kb":7744042,"num_write":1952007,"num_write_kb":7805803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83641,"ondisk_log_size":83641,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20353556,"kb_used_data":15607548,"kb_used_omap":4157,"kb_used_meta":4741826,"kb_avail":262761964,"statfs":{"total":289910292480,"available":269068251136,"internally_reserved":0,"allocated":15982129152,"data_stored":35583750223,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":4855630282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,4,7,3,11,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19857408,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4652,"num_read_kb":18568,"num_write":4808,"num_write_kb":19232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"2.7","version":"136'6554","reported_seq":9757,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.456813+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:08:19.456813+0000","last_peered":"2023-10-21T05:08:19.456813+0000","last_clean":"2023-10-21T05:08:19.456813+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:08:19.456813+0000","last_undegraded":"2023-10-21T05:08:19.456813+0000","last_fullsized":"2023-10-21T05:08:19.456813+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'3360","reported_seq":6994,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.442731+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:08:19.442731+0000","last_peered":"2023-10-21T05:08:19.442731+0000","last_clean":"2023-10-21T05:08:19.442731+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:08:19.442731+0000","last_undegraded":"2023-10-21T05:08:19.442731+0000","last_fullsized":"2023-10-21T05:08:19.442731+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3360,"log_dups_size":0,"ondisk_log_size":3360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'9780","reported_seq":14452,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.457119+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:08:19.457119+0000","last_peered":"2023-10-21T05:08:19.457119+0000","last_clean":"2023-10-21T05:08:19.457119+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:08:19.457119+0000","last_undegraded":"2023-10-21T05:08:19.457119+0000","last_fullsized":"2023-10-21T05:08:19.457119+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'6594","reported_seq":10811,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.456783+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:08:19.456783+0000","last_peered":"2023-10-21T05:08:19.456783+0000","last_clean":"2023-10-21T05:08:19.456783+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:08:19.456783+0000","last_undegraded":"2023-10-21T05:08:19.456783+0000","last_fullsized":"2023-10-21T05:08:19.456783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'488771","reported_seq":967598,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:18.958393+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:08:18.958393+0000","last_peered":"2023-10-21T05:08:18.958393+0000","last_clean":"2023-10-21T05:08:18.958393+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:08:18.958393+0000","last_undegraded":"2023-10-21T05:08:18.958393+0000","last_fullsized":"2023-10-21T05:08:18.958393+0000","mapping_epoch":14,"log_start":"135'478755","ondisk_log_start":"135'478755","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1225,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7160700928,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":483537,"num_read_kb":1907148,"num_write":482759,"num_write_kb":1931036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'13320","reported_seq":19694,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:18.782583+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:08:18.782583+0000","last_peered":"2023-10-21T05:08:18.782583+0000","last_clean":"2023-10-21T05:08:18.782583+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:08:18.782583+0000","last_undegraded":"2023-10-21T05:08:18.782583+0000","last_fullsized":"2023-10-21T05:08:18.782583+0000","mapping_epoch":11,"log_start":"70'3300","ondisk_log_start":"70'3300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'494037","reported_seq":979524,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:19.930162+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:08:19.930162+0000","last_peered":"2023-10-21T05:08:19.930162+0000","last_clean":"2023-10-21T05:08:19.930162+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:08:19.930162+0000","last_undegraded":"2023-10-21T05:08:19.930162+0000","last_fullsized":"2023-10-21T05:08:19.930162+0000","mapping_epoch":14,"log_start":"135'483958","ondisk_log_start":"135'483958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1017,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 104s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7248605184,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":490260,"num_read_kb":1933096,"num_write":488150,"num_write_kb":1952600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3335,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:14.340188+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:08:14.340188+0000","last_peered":"2023-10-21T05:08:14.340188+0000","last_clean":"2023-10-21T05:08:14.340188+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:08:14.340188+0000","last_undegraded":"2023-10-21T05:08:14.340188+0000","last_fullsized":"2023-10-21T05:08:14.340188+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'516565","reported_seq":1020164,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:18.983925+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:08:18.983925+0000","last_peered":"2023-10-21T05:08:18.983925+0000","last_clean":"2023-10-21T05:08:18.983925+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:08:18.983925+0000","last_undegraded":"2023-10-21T05:08:18.983925+0000","last_fullsized":"2023-10-21T05:08:18.983925+0000","mapping_epoch":14,"log_start":"135'506548","ondisk_log_start":"135'506548","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":856,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 118s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7572922368,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":508610,"num_read_kb":2006360,"num_write":510571,"num_write_kb":2042284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6647,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:14.342882+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:08:14.342882+0000","last_peered":"2023-10-21T05:08:14.342882+0000","last_clean":"2023-10-21T05:08:14.342882+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:08:14.342882+0000","last_undegraded":"2023-10-21T05:08:14.342882+0000","last_fullsized":"2023-10-21T05:08:14.342882+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'474238","reported_seq":940182,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.946996+0000","last_change":"2023-10-21T05:08:18.698978+0000","last_active":"2023-10-21T05:08:19.946996+0000","last_peered":"2023-10-21T05:08:19.946996+0000","last_clean":"2023-10-21T05:08:19.946996+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:08:19.946996+0000","last_undegraded":"2023-10-21T05:08:19.946996+0000","last_fullsized":"2023-10-21T05:08:19.946996+0000","mapping_epoch":14,"log_start":"135'464156","ondisk_log_start":"135'464156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6972149779,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":470577,"num_read_kb":1856368,"num_write":469369,"num_write_kb":1877470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'6517","reported_seq":9651,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.483621+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:08:19.483621+0000","last_peered":"2023-10-21T05:08:19.483621+0000","last_clean":"2023-10-21T05:08:19.483621+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:08:19.483621+0000","last_undegraded":"2023-10-21T05:08:19.483621+0000","last_fullsized":"2023-10-21T05:08:19.483621+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6517,"log_dups_size":0,"ondisk_log_size":6517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":308,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:07:29.449382+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:07:29.449382+0000","last_peered":"2023-10-21T05:07:29.449382+0000","last_clean":"2023-10-21T05:07:29.449382+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:07:29.449382+0000","last_undegraded":"2023-10-21T05:07:29.449382+0000","last_fullsized":"2023-10-21T05:07:29.449382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28954378259,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1952984,"num_read_kb":7702972,"num_write":1950849,"num_write_kb":7803390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15978500096,"data_stored":35582316582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40194,"ondisk_log_size":40194,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":43348,"ondisk_log_size":43348,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739195,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9060492,"kb_used_data":7801540,"kb_used_omap":2048,"kb_used_meta":1256895,"kb_avail":85311348,"statfs":{"total":96636764160,"available":87358820352,"internally_reserved":0,"allocated":7988776960,"data_stored":17789757424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1287060667},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.95699999999999996,"15min":1.0009999999999999},"min":{"1min":0.30499999999999999,"5min":0.185,"15min":0.184},"max":{"1min":3.1030000000000002,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":2.9060000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.98199999999999998,"15min":0.90200000000000002},"min":{"1min":0.34499999999999997,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":3.484,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":2.859}]},{"osd":1,"last update":"Sat Oct 21 05:08:00 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.383,"15min":1.133},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":3.1779999999999999,"5min":27.196999999999999,"15min":27.196999999999999},"last":5.3319999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.4119999999999999,"15min":1.2110000000000001},"min":{"1min":0.29099999999999998,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":3.0590000000000002,"5min":27.242999999999999,"15min":27.242999999999999},"last":5.367}]}]},{"osd":1,"up_from":8,"seq":34359739195,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5520864,"kb_used_data":3831640,"kb_used_omap":1037,"kb_used_meta":1688178,"kb_avail":88850976,"statfs":{"total":96636764160,"available":90983399424,"internally_reserved":0,"allocated":3923599360,"data_stored":8734746564,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1728695191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,10,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:49 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.81999999999999995,"15min":1.0129999999999999},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.8140000000000001,"5min":10.711,"15min":37.695999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.169,"15min":1.085},"min":{"1min":0.22,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":3.056,"5min":10.512,"15min":37.731000000000002},"last":0.36799999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:07:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.1739999999999999,"15min":1.1120000000000001},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":10.167999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.056,"5min":1.2170000000000001,"15min":1.1599999999999999},"min":{"1min":0.30299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":6},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739195,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5772200,"kb_used_data":3974368,"kb_used_omap":1071,"kb_used_meta":1796752,"kb_avail":88599640,"statfs":{"total":96636764160,"available":90726031360,"internally_reserved":0,"allocated":4069752832,"data_stored":9059246235,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1839874424},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.81899999999999995},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":3.1949999999999998,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.997,"15min":0.877},"min":{"1min":0.247,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.2440000000000002,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":2.117}]},{"osd":2,"last update":"Sat Oct 21 05:08:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.02,"15min":0.97999999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.4420000000000002,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.097,"15min":1.0740000000000001},"min":{"1min":0.224,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.1840000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":1.8939999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4068700160,"data_stored":9058746368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3921997824,"data_stored":8733859859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7987802112,"data_stored":17789710355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:08:22.287 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:22.287 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:22.287 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:22.288 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:22.288 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:42.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:08:42.833 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:08:42.833 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:08:42.855 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2206,"stamp":"2023-10-21T05:08:41.788053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28987791101,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2010171,"num_read_kb":7774882,"num_write":1959978,"num_write_kb":7837687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83752,"ondisk_log_size":83752,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20667268,"kb_used_data":15670376,"kb_used_omap":4157,"kb_used_meta":4992706,"kb_avail":262448252,"statfs":{"total":289910292480,"available":268747010048,"internally_reserved":0,"allocated":16046465024,"data_stored":35648086095,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":5112531402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,10,6,9,16,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16695296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3905,"num_read_kb":15584,"num_write":4045,"num_write_kb":16180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001715"},"pg_stats":[{"pgid":"2.7","version":"136'6570","reported_seq":9782,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.936745+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:08:39.936745+0000","last_peered":"2023-10-21T05:08:39.936745+0000","last_clean":"2023-10-21T05:08:39.936745+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:08:39.936745+0000","last_undegraded":"2023-10-21T05:08:39.936745+0000","last_fullsized":"2023-10-21T05:08:39.936745+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'3368","reported_seq":7008,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.383257+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:08:39.383257+0000","last_peered":"2023-10-21T05:08:39.383257+0000","last_clean":"2023-10-21T05:08:39.383257+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:08:39.383257+0000","last_undegraded":"2023-10-21T05:08:39.383257+0000","last_fullsized":"2023-10-21T05:08:39.383257+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'9804","reported_seq":14489,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.936852+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:08:39.936852+0000","last_peered":"2023-10-21T05:08:39.936852+0000","last_clean":"2023-10-21T05:08:39.936852+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:08:39.936852+0000","last_undegraded":"2023-10-21T05:08:39.936852+0000","last_fullsized":"2023-10-21T05:08:39.936852+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9804,"log_dups_size":0,"ondisk_log_size":9804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'6610","reported_seq":10836,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.936738+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:08:39.936738+0000","last_peered":"2023-10-21T05:08:39.936738+0000","last_clean":"2023-10-21T05:08:39.936738+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:08:39.936738+0000","last_undegraded":"2023-10-21T05:08:39.936738+0000","last_fullsized":"2023-10-21T05:08:39.936738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'490925","reported_seq":971720,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:39.209335+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:08:39.209335+0000","last_peered":"2023-10-21T05:08:39.209335+0000","last_clean":"2023-10-21T05:08:39.209335+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:08:39.209335+0000","last_undegraded":"2023-10-21T05:08:39.209335+0000","last_fullsized":"2023-10-21T05:08:39.209335+0000","mapping_epoch":14,"log_start":"135'480855","ondisk_log_start":"135'480855","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1396,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7169011712,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":485494,"num_read_kb":1914960,"num_write":484775,"num_write_kb":1939100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'13372","reported_seq":19762,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:34.817130+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:08:34.817130+0000","last_peered":"2023-10-21T05:08:34.817130+0000","last_clean":"2023-10-21T05:08:34.817130+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:08:34.817130+0000","last_undegraded":"2023-10-21T05:08:34.817130+0000","last_fullsized":"2023-10-21T05:08:34.817130+0000","mapping_epoch":11,"log_start":"70'3300","ondisk_log_start":"70'3300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'496167","reported_seq":983609,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:40.090025+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:08:40.090025+0000","last_peered":"2023-10-21T05:08:40.090025+0000","last_clean":"2023-10-21T05:08:40.090025+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:08:40.090025+0000","last_undegraded":"2023-10-21T05:08:40.090025+0000","last_fullsized":"2023-10-21T05:08:40.090025+0000","mapping_epoch":14,"log_start":"135'486158","ondisk_log_start":"135'486158","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1205,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 125s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7256788992,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":492202,"num_read_kb":1940840,"num_write":490129,"num_write_kb":1960516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3343,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:34.860871+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:08:34.860871+0000","last_peered":"2023-10-21T05:08:34.860871+0000","last_clean":"2023-10-21T05:08:34.860871+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:08:34.860871+0000","last_undegraded":"2023-10-21T05:08:34.860871+0000","last_fullsized":"2023-10-21T05:08:34.860871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'518759","reported_seq":1024341,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:39.209690+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:08:39.209690+0000","last_peered":"2023-10-21T05:08:39.209690+0000","last_clean":"2023-10-21T05:08:39.209690+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:08:39.209690+0000","last_undegraded":"2023-10-21T05:08:39.209690+0000","last_fullsized":"2023-10-21T05:08:39.209690+0000","mapping_epoch":14,"log_start":"135'508748","ondisk_log_start":"135'508748","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":999,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7581442048,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":510583,"num_read_kb":2014240,"num_write":512646,"num_write_kb":2050584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6663,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:34.818320+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:08:34.818320+0000","last_peered":"2023-10-21T05:08:34.818320+0000","last_clean":"2023-10-21T05:08:34.818320+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:08:34.818320+0000","last_undegraded":"2023-10-21T05:08:34.818320+0000","last_fullsized":"2023-10-21T05:08:34.818320+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'476139","reported_seq":943949,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:40.088447+0000","last_change":"2023-10-21T05:08:39.182404+0000","last_active":"2023-10-21T05:08:40.088447+0000","last_peered":"2023-10-21T05:08:40.088447+0000","last_clean":"2023-10-21T05:08:40.088447+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:08:40.088447+0000","last_undegraded":"2023-10-21T05:08:40.088447+0000","last_fullsized":"2023-10-21T05:08:40.088447+0000","mapping_epoch":14,"log_start":"135'466056","ondisk_log_start":"135'466056","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6979952659,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":472433,"num_read_kb":1863772,"num_write":471270,"num_write_kb":1885074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'6533","reported_seq":9676,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.937062+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:08:39.937062+0000","last_peered":"2023-10-21T05:08:39.937062+0000","last_clean":"2023-10-21T05:08:39.937062+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:08:39.937062+0000","last_undegraded":"2023-10-21T05:08:39.937062+0000","last_fullsized":"2023-10-21T05:08:39.937062+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6533,"log_dups_size":0,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":309,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.944838+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:08:19.944838+0000","last_peered":"2023-10-21T05:08:19.944838+0000","last_clean":"2023-10-21T05:08:19.944838+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:08:19.944838+0000","last_undegraded":"2023-10-21T05:08:19.944838+0000","last_fullsized":"2023-10-21T05:08:19.944838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28987195411,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1960712,"num_read_kb":7733812,"num_write":1958820,"num_write_kb":7835274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16043446272,"data_stored":35647262758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40173,"ondisk_log_size":40173,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":43480,"ondisk_log_size":43480,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9212420,"kb_used_data":7833084,"kb_used_omap":2048,"kb_used_meta":1377279,"kb_avail":85159420,"statfs":{"total":96636764160,"available":87203246080,"internally_reserved":0,"allocated":8021078016,"data_stored":17822058480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1410333883},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.95699999999999996,"15min":1.0009999999999999},"min":{"1min":0.30499999999999999,"5min":0.185,"15min":0.184},"max":{"1min":3.1030000000000002,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.317},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.98199999999999998,"15min":0.90200000000000002},"min":{"1min":0.34499999999999997,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":3.484,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:08:00 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.383,"15min":1.133},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":3.1779999999999999,"5min":27.196999999999999,"15min":27.196999999999999},"last":3.79},{"interface":"front","average":{"1min":1.171,"5min":1.4119999999999999,"15min":1.2110000000000001},"min":{"1min":0.29099999999999998,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":3.0590000000000002,"5min":27.242999999999999,"15min":27.242999999999999},"last":10.253}]}]},{"osd":1,"up_from":8,"seq":34359739199,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5590092,"kb_used_data":3846720,"kb_used_omap":1037,"kb_used_meta":1742322,"kb_avail":88781748,"statfs":{"total":96636764160,"available":90912509952,"internally_reserved":0,"allocated":3939041280,"data_stored":8750188484,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1784138647},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,1,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:49 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.81999999999999995,"15min":1.0129999999999999},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.8140000000000001,"5min":10.711,"15min":37.695999999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.169,"15min":1.085},"min":{"1min":0.22,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":3.056,"5min":10.512,"15min":37.731000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:07:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.1739999999999999,"15min":1.1120000000000001},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":10.167999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":2.7120000000000002},{"interface":"front","average":{"1min":1.056,"5min":1.2170000000000001,"15min":1.1599999999999999},"min":{"1min":0.30299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":6},"last":2.802}]}]},{"osd":0,"up_from":8,"seq":34359739199,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5864756,"kb_used_data":3990572,"kb_used_omap":1071,"kb_used_meta":1873104,"kb_avail":88507084,"statfs":{"total":96636764160,"available":90631254016,"internally_reserved":0,"allocated":4086345728,"data_stored":9075839131,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1918058872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,2,8,5,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.81899999999999995},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":3.1949999999999998,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.997,"15min":0.877},"min":{"1min":0.247,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.2440000000000002,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":1.3069999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:08:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.02,"15min":0.97999999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.4420000000000002,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.097,"15min":1.0740000000000001},"min":{"1min":0.224,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.1840000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4085436416,"data_stored":9075482624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3937894400,"data_stored":8749756435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8020115456,"data_stored":17822023699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:08:42.856 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:42.857 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:42.857 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:42.858 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:42.858 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:52.845 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:08:52.841+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:08:52.846 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:08:52.841+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:08:52.846 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:08:52.841+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:09:02.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:09:03.390 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:09:03.390 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:09:03.414 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2216,"stamp":"2023-10-21T05:09:01.791558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29020399357,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2018091,"num_read_kb":7806478,"num_write":1967913,"num_write_kb":7869427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83870,"ondisk_log_size":83870,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21003792,"kb_used_data":15734072,"kb_used_omap":4157,"kb_used_meta":5265538,"kb_avail":262111728,"statfs":{"total":289910292480,"available":268402409472,"internally_reserved":0,"allocated":16111689728,"data_stored":35713288533,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":5391911370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,3,5,14,21,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20586496,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5040,"num_read_kb":20100,"num_write":5000,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002114"},"pg_stats":[{"pgid":"2.7","version":"137'6598","reported_seq":9816,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:59.148434+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:08:59.148434+0000","last_peered":"2023-10-21T05:08:59.148434+0000","last_clean":"2023-10-21T05:08:59.148434+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:08:59.148434+0000","last_undegraded":"2023-10-21T05:08:59.148434+0000","last_fullsized":"2023-10-21T05:08:59.148434+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6598,"log_dups_size":0,"ondisk_log_size":6598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'3382","reported_seq":7025,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:57.956499+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:08:57.956499+0000","last_peered":"2023-10-21T05:08:57.956499+0000","last_clean":"2023-10-21T05:08:57.956499+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:08:57.956499+0000","last_undegraded":"2023-10-21T05:08:57.956499+0000","last_fullsized":"2023-10-21T05:08:57.956499+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'9846","reported_seq":14540,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:59.167445+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:08:59.167445+0000","last_peered":"2023-10-21T05:08:59.167445+0000","last_clean":"2023-10-21T05:08:59.167445+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:08:59.167445+0000","last_undegraded":"2023-10-21T05:08:59.167445+0000","last_fullsized":"2023-10-21T05:08:59.167445+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'6638","reported_seq":10870,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:59.156720+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:08:59.156720+0000","last_peered":"2023-10-21T05:08:59.156720+0000","last_clean":"2023-10-21T05:08:59.156720+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:08:59.156720+0000","last_undegraded":"2023-10-21T05:08:59.156720+0000","last_fullsized":"2023-10-21T05:08:59.156720+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6638,"log_dups_size":0,"ondisk_log_size":6638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'493064","reported_seq":975829,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:59.348927+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:08:59.348927+0000","last_peered":"2023-10-21T05:08:59.348927+0000","last_clean":"2023-10-21T05:08:59.348927+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:08:59.348927+0000","last_undegraded":"2023-10-21T05:08:59.348927+0000","last_fullsized":"2023-10-21T05:08:59.348927+0000","mapping_epoch":14,"log_start":"135'483057","ondisk_log_start":"135'483057","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1597,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 161s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7177097216,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":487452,"num_read_kb":1922772,"num_write":486748,"num_write_kb":1946992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"137'13404","reported_seq":19811,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:58.397313+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:08:58.397313+0000","last_peered":"2023-10-21T05:08:58.397313+0000","last_clean":"2023-10-21T05:08:58.397313+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:08:58.397313+0000","last_undegraded":"2023-10-21T05:08:58.397313+0000","last_fullsized":"2023-10-21T05:08:58.397313+0000","mapping_epoch":11,"log_start":"70'3400","ondisk_log_start":"70'3400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'498292","reported_seq":987672,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:00.312199+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:09:00.312199+0000","last_peered":"2023-10-21T05:09:00.312199+0000","last_clean":"2023-10-21T05:09:00.312199+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:09:00.312199+0000","last_undegraded":"2023-10-21T05:09:00.312199+0000","last_fullsized":"2023-10-21T05:09:00.312199+0000","mapping_epoch":14,"log_start":"135'488258","ondisk_log_start":"135'488258","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1363,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 145s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7264985088,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":494128,"num_read_kb":1948536,"num_write":492128,"num_write_kb":1968512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3353,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:55.297988+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:08:55.297988+0000","last_peered":"2023-10-21T05:08:55.297988+0000","last_clean":"2023-10-21T05:08:55.297988+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:08:55.297988+0000","last_undegraded":"2023-10-21T05:08:55.297988+0000","last_fullsized":"2023-10-21T05:08:55.297988+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'520941","reported_seq":1028631,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:59.358475+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:08:59.358475+0000","last_peered":"2023-10-21T05:08:59.358475+0000","last_clean":"2023-10-21T05:08:59.358475+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:08:59.358475+0000","last_undegraded":"2023-10-21T05:08:59.358475+0000","last_fullsized":"2023-10-21T05:08:59.358475+0000","mapping_epoch":14,"log_start":"135'510848","ondisk_log_start":"135'510848","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1148,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 158s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7589986304,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":512678,"num_read_kb":2022600,"num_write":514707,"num_write_kb":2058828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6680,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:55.306805+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:08:55.306805+0000","last_peered":"2023-10-21T05:08:55.306805+0000","last_clean":"2023-10-21T05:08:55.306805+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:08:55.306805+0000","last_undegraded":"2023-10-21T05:08:55.306805+0000","last_fullsized":"2023-10-21T05:08:55.306805+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'478041","reported_seq":947796,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:00.282519+0000","last_change":"2023-10-21T05:08:57.421831+0000","last_active":"2023-10-21T05:09:00.282519+0000","last_peered":"2023-10-21T05:09:00.282519+0000","last_clean":"2023-10-21T05:09:00.282519+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:09:00.282519+0000","last_undegraded":"2023-10-21T05:09:00.282519+0000","last_fullsized":"2023-10-21T05:09:00.282519+0000","mapping_epoch":14,"log_start":"135'467956","ondisk_log_start":"135'467956","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6987735059,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":474374,"num_read_kb":1871500,"num_write":473172,"num_write_kb":1892682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'6561","reported_seq":9711,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:59.167304+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:08:59.167304+0000","last_peered":"2023-10-21T05:08:59.167304+0000","last_clean":"2023-10-21T05:08:59.167304+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:08:59.167304+0000","last_undegraded":"2023-10-21T05:08:59.167304+0000","last_fullsized":"2023-10-21T05:08:59.167304+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":309,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.944838+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:08:19.944838+0000","last_peered":"2023-10-21T05:08:19.944838+0000","last_clean":"2023-10-21T05:08:19.944838+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:08:19.944838+0000","last_undegraded":"2023-10-21T05:08:19.944838+0000","last_fullsized":"2023-10-21T05:08:19.944838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29019803667,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1968632,"num_read_kb":7765408,"num_write":1966755,"num_write_kb":7867014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16108425216,"data_stored":35712241702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40219,"ondisk_log_size":40219,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":43552,"ondisk_log_size":43552,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739203,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9408916,"kb_used_data":7865036,"kb_used_omap":2048,"kb_used_meta":1541823,"kb_avail":84962924,"statfs":{"total":96636764160,"available":87002034176,"internally_reserved":0,"allocated":8053796864,"data_stored":17854769906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1578826939},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.95699999999999996,"15min":1.0009999999999999},"min":{"1min":0.30499999999999999,"5min":0.185,"15min":0.184},"max":{"1min":3.1030000000000002,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":1.48},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.98199999999999998,"15min":0.90200000000000002},"min":{"1min":0.34499999999999997,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":3.484,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":1.4350000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:08:00 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.383,"15min":1.133},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":3.1779999999999999,"5min":27.196999999999999,"15min":27.196999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.4119999999999999,"15min":1.2110000000000001},"min":{"1min":0.29099999999999998,"5min":0.23300000000000001,"15min":0.222},"max":{"1min":3.0590000000000002,"5min":27.242999999999999,"15min":27.242999999999999},"last":2.8610000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739203,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5659772,"kb_used_data":3862260,"kb_used_omap":1037,"kb_used_meta":1796466,"kb_avail":88712068,"statfs":{"total":96636764160,"available":90841157632,"internally_reserved":0,"allocated":3954954240,"data_stored":8766094022,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1839582103},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,8,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81799999999999995,"15min":1.002},"min":{"1min":0.21299999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":6.2140000000000004,"5min":10.711,"15min":37.695999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.0489999999999999,"15min":1.087},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":1.7929999999999999,"5min":10.512,"15min":37.731000000000002},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.085,"15min":1.079},"min":{"1min":0.24199999999999999,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":2.7120000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.127,"15min":1.1519999999999999},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":2.802,"5min":4.8650000000000002,"15min":6},"last":0.66400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739203,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5935104,"kb_used_data":4006776,"kb_used_omap":1071,"kb_used_meta":1927248,"kb_avail":88436736,"statfs":{"total":96636764160,"available":90559217664,"internally_reserved":0,"allocated":4102938624,"data_stored":9092424605,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":1973502328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.81899999999999995},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":3.1949999999999998,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":0.245},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.997,"15min":0.877},"min":{"1min":0.247,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.2440000000000002,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.27000000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:08:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.02,"15min":0.97999999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.4420000000000002,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.997},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.097,"15min":1.0740000000000001},"min":{"1min":0.224,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.1840000000000002,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4101959680,"data_stored":9092005888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3953864704,"data_stored":8765726739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8052600832,"data_stored":17854509075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:09:03.416 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:03.416 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:03.416 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:03.417 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:03.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:09:03.955 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:09:03.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:09:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:09:04.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:09:05.084 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:09:05.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:09:05.653 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:09:05.679 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:09:20.918 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:20.915+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:09:24.947 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:24.945+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:09:24.948 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:24.945+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:09:24.948 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:24.945+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:09:24.948 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:24.945+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:09:25.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:09:26.210 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:09:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:09:26.232 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2228,"stamp":"2023-10-21T05:09:25.795993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29060945661,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2027475,"num_read_kb":7843962,"num_write":1977800,"num_write_kb":7908975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84259,"ondisk_log_size":84259,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21399588,"kb_used_data":15812620,"kb_used_omap":4157,"kb_used_meta":5582786,"kb_avail":261715932,"statfs":{"total":289910292480,"available":267997114368,"internally_reserved":0,"allocated":16192122880,"data_stored":35793721685,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":5716773322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,12,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24313856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5575,"num_read_kb":22260,"num_write":5929,"num_write_kb":23716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002361"},"pg_stats":[{"pgid":"2.7","version":"137'6636","reported_seq":9864,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:24.521226+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:09:24.521226+0000","last_peered":"2023-10-21T05:09:24.521226+0000","last_clean":"2023-10-21T05:09:24.521226+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:09:24.521226+0000","last_undegraded":"2023-10-21T05:09:24.521226+0000","last_fullsized":"2023-10-21T05:09:24.521226+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'3401","reported_seq":7049,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:23.027378+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:09:23.027378+0000","last_peered":"2023-10-21T05:09:23.027378+0000","last_clean":"2023-10-21T05:09:23.027378+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:09:23.027378+0000","last_undegraded":"2023-10-21T05:09:23.027378+0000","last_fullsized":"2023-10-21T05:09:23.027378+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3401,"log_dups_size":0,"ondisk_log_size":3401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'9903","reported_seq":14613,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:24.532813+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:09:24.532813+0000","last_peered":"2023-10-21T05:09:24.532813+0000","last_clean":"2023-10-21T05:09:24.532813+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:09:24.532813+0000","last_undegraded":"2023-10-21T05:09:24.532813+0000","last_fullsized":"2023-10-21T05:09:24.532813+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9903,"log_dups_size":0,"ondisk_log_size":9903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'6676","reported_seq":10919,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:24.521383+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:09:24.521383+0000","last_peered":"2023-10-21T05:09:24.521383+0000","last_clean":"2023-10-21T05:09:24.521383+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:09:24.521383+0000","last_undegraded":"2023-10-21T05:09:24.521383+0000","last_fullsized":"2023-10-21T05:09:24.521383+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'495645","reported_seq":980730,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:24.703826+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:09:24.703826+0000","last_peered":"2023-10-21T05:09:24.703826+0000","last_clean":"2023-10-21T05:09:24.703826+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:09:24.703826+0000","last_undegraded":"2023-10-21T05:09:24.703826+0000","last_fullsized":"2023-10-21T05:09:24.703826+0000","mapping_epoch":14,"log_start":"136'485557","ondisk_log_start":"136'485557","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1786,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 186s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7187021824,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":489755,"num_read_kb":1931984,"num_write":489173,"num_write_kb":1956692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"137'13464","reported_seq":19892,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:21.401884+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:09:21.401884+0000","last_peered":"2023-10-21T05:09:21.401884+0000","last_clean":"2023-10-21T05:09:21.401884+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:09:21.401884+0000","last_undegraded":"2023-10-21T05:09:21.401884+0000","last_fullsized":"2023-10-21T05:09:21.401884+0000","mapping_epoch":11,"log_start":"70'3400","ondisk_log_start":"70'3400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'500942","reported_seq":992799,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:25.517498+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:09:25.517498+0000","last_peered":"2023-10-21T05:09:25.517498+0000","last_clean":"2023-10-21T05:09:25.517498+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:09:25.517498+0000","last_undegraded":"2023-10-21T05:09:25.517498+0000","last_fullsized":"2023-10-21T05:09:25.517498+0000","mapping_epoch":14,"log_start":"136'490858","ondisk_log_start":"136'490858","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1492,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 170s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7275433984,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":496586,"num_read_kb":1958340,"num_write":494671,"num_write_kb":1978684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3363,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:20.902208+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:09:20.902208+0000","last_peered":"2023-10-21T05:09:20.902208+0000","last_clean":"2023-10-21T05:09:20.902208+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:09:20.902208+0000","last_undegraded":"2023-10-21T05:09:20.902208+0000","last_fullsized":"2023-10-21T05:09:20.902208+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'523648","reported_seq":1033755,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:24.694698+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:09:24.694698+0000","last_peered":"2023-10-21T05:09:24.694698+0000","last_clean":"2023-10-21T05:09:24.694698+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:09:24.694698+0000","last_undegraded":"2023-10-21T05:09:24.694698+0000","last_fullsized":"2023-10-21T05:09:24.694698+0000","mapping_epoch":14,"log_start":"136'513548","ondisk_log_start":"136'513548","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1379,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7600312320,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":515081,"num_read_kb":2032196,"num_write":517225,"num_write_kb":2068900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6700,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:20.901573+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:09:20.901573+0000","last_peered":"2023-10-21T05:09:20.901573+0000","last_clean":"2023-10-21T05:09:20.901573+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:09:20.901573+0000","last_undegraded":"2023-10-21T05:09:20.901573+0000","last_fullsized":"2023-10-21T05:09:20.901573+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'480442","reported_seq":952424,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:25.538204+0000","last_change":"2023-10-21T05:09:23.598259+0000","last_active":"2023-10-21T05:09:25.538204+0000","last_peered":"2023-10-21T05:09:25.538204+0000","last_clean":"2023-10-21T05:09:25.538204+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:09:25.538204+0000","last_undegraded":"2023-10-21T05:09:25.538204+0000","last_fullsized":"2023-10-21T05:09:25.538204+0000","mapping_epoch":14,"log_start":"136'470356","ondisk_log_start":"136'470356","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":6997581843,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":476594,"num_read_kb":1880372,"num_write":475573,"num_write_kb":1902286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'6599","reported_seq":9759,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:24.561362+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:09:24.561362+0000","last_peered":"2023-10-21T05:09:24.561362+0000","last_clean":"2023-10-21T05:09:24.561362+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:09:24.561362+0000","last_undegraded":"2023-10-21T05:09:24.561362+0000","last_fullsized":"2023-10-21T05:09:24.561362+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":309,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.944838+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:08:19.944838+0000","last_peered":"2023-10-21T05:08:19.944838+0000","last_clean":"2023-10-21T05:08:19.944838+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:08:19.944838+0000","last_undegraded":"2023-10-21T05:08:19.944838+0000","last_fullsized":"2023-10-21T05:08:19.944838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29060349971,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1978016,"num_read_kb":7802892,"num_write":1976642,"num_write_kb":7906562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16189083648,"data_stored":35792900134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40358,"ondisk_log_size":40358,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":43802,"ondisk_log_size":43802,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739208,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9596428,"kb_used_data":7904004,"kb_used_omap":2048,"kb_used_meta":1690367,"kb_avail":84775412,"statfs":{"total":96636764160,"available":86810021888,"internally_reserved":0,"allocated":8093700096,"data_stored":17894673138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1730935995},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.0449999999999999,"15min":1.024},"min":{"1min":0.249,"5min":0.185,"15min":0.184},"max":{"1min":3.8170000000000002,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.97999999999999998,"15min":0.91500000000000004},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.859,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.80300000000000005}]},{"osd":1,"last update":"Sat Oct 21 05:09:03 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4970000000000001,"15min":1.1799999999999999},"min":{"1min":0.27900000000000003,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":5.3319999999999999,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.498},{"interface":"front","average":{"1min":1.9179999999999999,"5min":1.5920000000000001,"15min":1.2749999999999999},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":10.253,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359739208,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5755956,"kb_used_data":3882092,"kb_used_omap":1037,"kb_used_meta":1872818,"kb_avail":88615884,"statfs":{"total":96636764160,"available":90742665216,"internally_reserved":0,"allocated":3975262208,"data_stored":8786401990,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1917766551},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81799999999999995,"15min":1.002},"min":{"1min":0.21299999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":6.2140000000000004,"5min":10.711,"15min":37.695999999999998},"last":2.6509999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.0489999999999999,"15min":1.087},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":1.7929999999999999,"5min":10.512,"15min":37.731000000000002},"last":2.5449999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.085,"15min":1.079},"min":{"1min":0.24199999999999999,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":2.7120000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":3},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.127,"15min":1.1519999999999999},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":2.802,"5min":4.8650000000000002,"15min":6},"last":2.613}]}]},{"osd":0,"up_from":8,"seq":34359739208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6047204,"kb_used_data":4026524,"kb_used_omap":1071,"kb_used_meta":2019600,"kb_avail":88324636,"statfs":{"total":96636764160,"available":90444427264,"internally_reserved":0,"allocated":4123160576,"data_stored":9112646557,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":2068070776},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.88400000000000001,"15min":0.82599999999999996},"min":{"1min":0.23499999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":3.637,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":1.978},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.002,"15min":0.89000000000000001},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.1289999999999996,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":0.37}]},{"osd":2,"last update":"Sat Oct 21 05:09:07 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.98399999999999999,"15min":0.98699999999999999},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":4.907,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.0049999999999999,"15min":1.0720000000000001},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.929,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4122206208,"data_stored":9112252416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3974127616,"data_stored":8785989651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8092749824,"data_stored":17894658067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:09:26.233 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:26.234 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:26.234 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:26.235 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:26.235 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:09:30.244 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:30.241+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:09:46.236 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:09:46.778 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:09:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:09:46.799 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2238,"stamp":"2023-10-21T05:09:45.799520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29092476669,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2035361,"num_read_kb":7875414,"num_write":1985482,"num_write_kb":7939703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84118,"ondisk_log_size":84118,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21759240,"kb_used_data":15875176,"kb_used_omap":4157,"kb_used_meta":5879874,"kb_avail":261356280,"statfs":{"total":289910292480,"available":267628830720,"internally_reserved":0,"allocated":16256180224,"data_stored":35857779029,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":6020991434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,8,13,14,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23834624,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5950,"num_read_kb":23756,"num_write":5795,"num_write_kb":23180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002041"},"pg_stats":[{"pgid":"2.7","version":"137'6654","reported_seq":9891,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:45.279503+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:09:45.279503+0000","last_peered":"2023-10-21T05:09:45.279503+0000","last_clean":"2023-10-21T05:09:45.279503+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:09:45.279503+0000","last_undegraded":"2023-10-21T05:09:45.279503+0000","last_fullsized":"2023-10-21T05:09:45.279503+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6654,"log_dups_size":0,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'3410","reported_seq":7062,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:44.227540+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:09:44.227540+0000","last_peered":"2023-10-21T05:09:44.227540+0000","last_clean":"2023-10-21T05:09:44.227540+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:09:44.227540+0000","last_undegraded":"2023-10-21T05:09:44.227540+0000","last_fullsized":"2023-10-21T05:09:44.227540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3410,"log_dups_size":0,"ondisk_log_size":3410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'9930","reported_seq":14652,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:45.285721+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:09:45.285721+0000","last_peered":"2023-10-21T05:09:45.285721+0000","last_clean":"2023-10-21T05:09:45.285721+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:09:45.285721+0000","last_undegraded":"2023-10-21T05:09:45.285721+0000","last_fullsized":"2023-10-21T05:09:45.285721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9930,"log_dups_size":0,"ondisk_log_size":9930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'6694","reported_seq":10945,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:45.279323+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:09:45.279323+0000","last_peered":"2023-10-21T05:09:45.279323+0000","last_clean":"2023-10-21T05:09:45.279323+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:09:45.279323+0000","last_undegraded":"2023-10-21T05:09:45.279323+0000","last_fullsized":"2023-10-21T05:09:45.279323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'497660","reported_seq":984755,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:44.974624+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:09:44.974624+0000","last_peered":"2023-10-21T05:09:44.974624+0000","last_clean":"2023-10-21T05:09:44.974624+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:09:44.974624+0000","last_undegraded":"2023-10-21T05:09:44.974624+0000","last_fullsized":"2023-10-21T05:09:44.974624+0000","mapping_epoch":14,"log_start":"136'487657","ondisk_log_start":"136'487657","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2001,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 207s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7194558464,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":491747,"num_read_kb":1939924,"num_write":491011,"num_write_kb":1964044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"137'13525","reported_seq":19969,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:44.033760+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:09:44.033760+0000","last_peered":"2023-10-21T05:09:44.033760+0000","last_clean":"2023-10-21T05:09:44.033760+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:09:44.033760+0000","last_undegraded":"2023-10-21T05:09:44.033760+0000","last_fullsized":"2023-10-21T05:09:44.033760+0000","mapping_epoch":11,"log_start":"72'3500","ondisk_log_start":"72'3500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'503042","reported_seq":996851,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:45.718421+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:09:45.718421+0000","last_peered":"2023-10-21T05:09:45.718421+0000","last_clean":"2023-10-21T05:09:45.718421+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:09:45.718421+0000","last_undegraded":"2023-10-21T05:09:45.718421+0000","last_fullsized":"2023-10-21T05:09:45.718421+0000","mapping_epoch":14,"log_start":"136'492958","ondisk_log_start":"136'492958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1622,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 190s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7283658752,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":498528,"num_read_kb":1966096,"num_write":496666,"num_write_kb":1986664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3371,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:41.387574+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:09:41.387574+0000","last_peered":"2023-10-21T05:09:41.387574+0000","last_clean":"2023-10-21T05:09:41.387574+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:09:41.387574+0000","last_undegraded":"2023-10-21T05:09:41.387574+0000","last_fullsized":"2023-10-21T05:09:41.387574+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'525817","reported_seq":1038044,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:44.972236+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:09:44.972236+0000","last_peered":"2023-10-21T05:09:44.972236+0000","last_clean":"2023-10-21T05:09:44.972236+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:09:44.972236+0000","last_undegraded":"2023-10-21T05:09:44.972236+0000","last_fullsized":"2023-10-21T05:09:44.972236+0000","mapping_epoch":14,"log_start":"136'515748","ondisk_log_start":"136'515748","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1559,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 204s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7608598528,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":517186,"num_read_kb":2040580,"num_write":519250,"num_write_kb":2077000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6716,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:41.377602+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:09:41.377602+0000","last_peered":"2023-10-21T05:09:41.377602+0000","last_clean":"2023-10-21T05:09:41.377602+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:09:41.377602+0000","last_undegraded":"2023-10-21T05:09:41.377602+0000","last_fullsized":"2023-10-21T05:09:41.377602+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'482266","reported_seq":956102,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:45.709133+0000","last_change":"2023-10-21T05:09:40.767399+0000","last_active":"2023-10-21T05:09:45.709133+0000","last_peered":"2023-10-21T05:09:45.709133+0000","last_clean":"2023-10-21T05:09:45.709133+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:09:45.709133+0000","last_undegraded":"2023-10-21T05:09:45.709133+0000","last_fullsized":"2023-10-21T05:09:45.709133+0000","mapping_epoch":14,"log_start":"136'472256","ondisk_log_start":"136'472256","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7005065235,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":478441,"num_read_kb":1887744,"num_write":477397,"num_write_kb":1909582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'6617","reported_seq":9785,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:45.276537+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:09:45.276537+0000","last_peered":"2023-10-21T05:09:45.276537+0000","last_clean":"2023-10-21T05:09:45.276537+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:09:45.276537+0000","last_undegraded":"2023-10-21T05:09:45.276537+0000","last_fullsized":"2023-10-21T05:09:45.276537+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":309,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-21T05:08:19.944838+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:08:19.944838+0000","last_peered":"2023-10-21T05:08:19.944838+0000","last_clean":"2023-10-21T05:08:19.944838+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:08:19.944838+0000","last_undegraded":"2023-10-21T05:08:19.944838+0000","last_fullsized":"2023-10-21T05:08:19.944838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29091880979,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1985902,"num_read_kb":7834344,"num_write":1984324,"num_write_kb":7937290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16251977728,"data_stored":35855794214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40166,"ondisk_log_size":40166,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":43853,"ondisk_log_size":43853,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739212,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9779548,"kb_used_data":7934416,"kb_used_omap":2048,"kb_used_meta":1843071,"kb_avail":84592292,"statfs":{"total":96636764160,"available":86622507008,"internally_reserved":0,"allocated":8124841984,"data_stored":17925815026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1887304891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.0449999999999999,"15min":1.024},"min":{"1min":0.249,"5min":0.185,"15min":0.184},"max":{"1min":3.8170000000000002,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":1.091},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.97999999999999998,"15min":0.91500000000000004},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.859,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":1.048}]},{"osd":1,"last update":"Sat Oct 21 05:09:03 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4970000000000001,"15min":1.1799999999999999},"min":{"1min":0.27900000000000003,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":5.3319999999999999,"5min":27.196999999999999,"15min":27.196999999999999},"last":3.0150000000000001},{"interface":"front","average":{"1min":1.9179999999999999,"5min":1.5920000000000001,"15min":1.2749999999999999},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":10.253,"5min":27.242999999999999,"15min":27.242999999999999},"last":0.42699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739212,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5844112,"kb_used_data":3898056,"kb_used_omap":1037,"kb_used_meta":1945010,"kb_avail":88527728,"statfs":{"total":96636764160,"available":90652393472,"internally_reserved":0,"allocated":3991609344,"data_stored":8802749126,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":1991691159},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,6,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81799999999999995,"15min":1.002},"min":{"1min":0.21299999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":6.2140000000000004,"5min":10.711,"15min":37.695999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.0489999999999999,"15min":1.087},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":1.7929999999999999,"5min":10.512,"15min":37.731000000000002},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.085,"15min":1.079},"min":{"1min":0.24199999999999999,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":2.7120000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.127,"15min":1.1519999999999999},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":2.802,"5min":4.8650000000000002,"15min":6},"last":0.71699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6135580,"kb_used_data":4042704,"kb_used_omap":1071,"kb_used_meta":2091792,"kb_avail":88236260,"statfs":{"total":96636764160,"available":90353930240,"internally_reserved":0,"allocated":4139728896,"data_stored":9129214877,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":2141995384},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1,8,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.88400000000000001,"15min":0.82599999999999996},"min":{"1min":0.23499999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":3.637,"5min":4.6139999999999999,"15min":7.7140000000000004},"last":4.3170000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.002,"15min":0.89000000000000001},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.1289999999999996,"5min":4.6500000000000004,"15min":7.8559999999999999},"last":4.3540000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:09:07 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.98399999999999999,"15min":0.98699999999999999},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":4.907,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.0049999999999999,"15min":1.0720000000000001},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.929,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":5.133}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4138029056,"data_stored":9128075264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3989757952,"data_stored":8801619987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8124190720,"data_stored":17926098963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:09:46.801 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:46.801 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:46.801 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:46.802 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:46.802 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:09:49.550 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:49.547+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:09:49.551 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:49.547+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:09:49.552 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:09:49.547+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:10:06.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:07.334 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:10:07.335 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:10:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2249,"stamp":"2023-10-21T05:10:05.803161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29120345853,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2042061,"num_read_kb":7902170,"num_write":1992290,"num_write_kb":7966935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84436,"ondisk_log_size":84436,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21960640,"kb_used_data":15931552,"kb_used_omap":4157,"kb_used_meta":6024898,"kb_avail":261154880,"statfs":{"total":289910292480,"available":267422597120,"internally_reserved":0,"allocated":16313909248,"data_stored":35915485658,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4257334,"internal_metadata":6169496010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,3,7,10,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12099584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2938,"num_read_kb":11720,"num_write":2956,"num_write_kb":11824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001710"},"pg_stats":[{"pgid":"2.7","version":"137'6676","reported_seq":9920,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:58.699644+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:09:58.699644+0000","last_peered":"2023-10-21T05:09:58.699644+0000","last_clean":"2023-10-21T05:09:58.699644+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:09:58.699644+0000","last_undegraded":"2023-10-21T05:09:58.699644+0000","last_fullsized":"2023-10-21T05:09:58.699644+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'3421","reported_seq":7077,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:58.692073+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:09:58.692073+0000","last_peered":"2023-10-21T05:09:58.692073+0000","last_clean":"2023-10-21T05:09:58.692073+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:09:58.692073+0000","last_undegraded":"2023-10-21T05:09:58.692073+0000","last_fullsized":"2023-10-21T05:09:58.692073+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'9963","reported_seq":14695,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:58.709116+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:09:58.709116+0000","last_peered":"2023-10-21T05:09:58.709116+0000","last_clean":"2023-10-21T05:09:58.709116+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:09:58.709116+0000","last_undegraded":"2023-10-21T05:09:58.709116+0000","last_fullsized":"2023-10-21T05:09:58.709116+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'6716","reported_seq":10974,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:58.697093+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:09:58.697093+0000","last_peered":"2023-10-21T05:09:58.697093+0000","last_clean":"2023-10-21T05:09:58.697093+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:09:58.697093+0000","last_undegraded":"2023-10-21T05:09:58.697093+0000","last_fullsized":"2023-10-21T05:09:58.697093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'499737","reported_seq":988712,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:05.264149+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:10:05.264149+0000","last_peered":"2023-10-21T05:10:05.264149+0000","last_clean":"2023-10-21T05:10:05.264149+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:10:05.264149+0000","last_undegraded":"2023-10-21T05:10:05.264149+0000","last_fullsized":"2023-10-21T05:10:05.264149+0000","mapping_epoch":14,"log_start":"136'489657","ondisk_log_start":"136'489657","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2141,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 227s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7202586624,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":493615,"num_read_kb":1947384,"num_write":492971,"num_write_kb":1971884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'13585","reported_seq":20046,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:04.731338+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:10:04.731338+0000","last_peered":"2023-10-21T05:10:04.731338+0000","last_clean":"2023-10-21T05:10:04.731338+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:10:04.731338+0000","last_undegraded":"2023-10-21T05:10:04.731338+0000","last_fullsized":"2023-10-21T05:10:04.731338+0000","mapping_epoch":11,"log_start":"72'3500","ondisk_log_start":"72'3500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'504608","reported_seq":999884,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:00.786093+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:10:00.786093+0000","last_peered":"2023-10-21T05:10:00.786093+0000","last_clean":"2023-10-21T05:10:00.786093+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:10:00.786093+0000","last_undegraded":"2023-10-21T05:10:00.786093+0000","last_fullsized":"2023-10-21T05:10:00.786093+0000","mapping_epoch":14,"log_start":"136'494558","ondisk_log_start":"136'494558","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1722,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 205s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7289729024,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":499981,"num_read_kb":1971904,"num_write":498150,"num_write_kb":1992600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3380,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:01.858194+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:10:01.858194+0000","last_peered":"2023-10-21T05:10:01.858194+0000","last_clean":"2023-10-21T05:10:01.858194+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:10:01.858194+0000","last_undegraded":"2023-10-21T05:10:01.858194+0000","last_fullsized":"2023-10-21T05:10:01.858194+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'527945","reported_seq":1042226,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:05.254306+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:10:05.254306+0000","last_peered":"2023-10-21T05:10:05.254306+0000","last_clean":"2023-10-21T05:10:05.254306+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:10:05.254306+0000","last_undegraded":"2023-10-21T05:10:05.254306+0000","last_fullsized":"2023-10-21T05:10:05.254306+0000","mapping_epoch":14,"log_start":"136'517848","ondisk_log_start":"136'517848","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1731,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 224s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7616729088,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":519226,"num_read_kb":2048724,"num_write":521237,"num_write_kb":2084948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6733,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:01.867497+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:10:01.867497+0000","last_peered":"2023-10-21T05:10:01.867497+0000","last_clean":"2023-10-21T05:10:01.867497+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:10:01.867497+0000","last_undegraded":"2023-10-21T05:10:01.867497+0000","last_fullsized":"2023-10-21T05:10:01.867497+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'483643","reported_seq":958822,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:00.819665+0000","last_change":"2023-10-21T05:09:57.949095+0000","last_active":"2023-10-21T05:10:00.819665+0000","last_peered":"2023-10-21T05:10:00.819665+0000","last_clean":"2023-10-21T05:10:00.819665+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:10:00.819665+0000","last_undegraded":"2023-10-21T05:10:00.819665+0000","last_fullsized":"2023-10-21T05:10:00.819665+0000","mapping_epoch":14,"log_start":"136'473556","ondisk_log_start":"136'473556","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7010705427,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":479780,"num_read_kb":1893088,"num_write":478774,"num_write_kb":1915090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'6639","reported_seq":9814,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:09:58.741241+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:09:58.741241+0000","last_peered":"2023-10-21T05:09:58.741241+0000","last_clean":"2023-10-21T05:09:58.741241+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:09:58.741241+0000","last_undegraded":"2023-10-21T05:09:58.741241+0000","last_fullsized":"2023-10-21T05:09:58.741241+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":310,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.389654+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:08:39.389654+0000","last_peered":"2023-10-21T05:08:39.389654+0000","last_clean":"2023-10-21T05:08:39.389654+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:08:39.389654+0000","last_undegraded":"2023-10-21T05:08:39.389654+0000","last_fullsized":"2023-10-21T05:08:39.389654+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29119750163,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":1992602,"num_read_kb":7861100,"num_write":1991132,"num_write_kb":7964522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16311635968,"data_stored":35915452454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40314,"ondisk_log_size":40314,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44023,"ondisk_log_size":44023,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9901208,"kb_used_data":7964940,"kb_used_omap":2048,"kb_used_meta":1934207,"kb_avail":84470632,"statfs":{"total":96636764160,"available":86497927168,"internally_reserved":0,"allocated":8156098560,"data_stored":17957064137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":1980628155},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.0449999999999999,"15min":1.024},"min":{"1min":0.249,"5min":0.185,"15min":0.184},"max":{"1min":3.8170000000000002,"5min":4.2910000000000004,"15min":8.7509999999999994},"last":1.181},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.97999999999999998,"15min":0.91500000000000004},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.859,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":1.6619999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:09:03 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4970000000000001,"15min":1.1799999999999999},"min":{"1min":0.27900000000000003,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":5.3319999999999999,"5min":27.196999999999999,"15min":27.196999999999999},"last":0.995},{"interface":"front","average":{"1min":1.9179999999999999,"5min":1.5920000000000001,"15min":1.2749999999999999},"min":{"1min":0.29699999999999999,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":10.253,"5min":27.242999999999999,"15min":27.242999999999999},"last":1.0309999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739215,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5882648,"kb_used_data":3908812,"kb_used_omap":1037,"kb_used_meta":1972786,"kb_avail":88489192,"statfs":{"total":96636764160,"available":90612932608,"internally_reserved":0,"allocated":4002623488,"data_stored":8813755805,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":2020133783},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,0,1,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.90200000000000002,"15min":0.90800000000000003},"min":{"1min":0.218,"5min":0.186,"15min":0.16400000000000001},"max":{"1min":4.8929999999999998,"5min":10.711,"15min":10.711},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.054,"15min":0.98099999999999998},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":2.5449999999999999,"5min":10.512,"15min":10.512},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:09:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.05,"15min":1.0700000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":3,"5min":10.167999999999999,"15min":10.167999999999999},"last":1.133},{"interface":"front","average":{"1min":1.004,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.30299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":2.8490000000000002,"5min":4.8650000000000002,"15min":6},"last":1.0860000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6176784,"kb_used_data":4057800,"kb_used_omap":1071,"kb_used_meta":2117904,"kb_avail":88195056,"statfs":{"total":96636764160,"available":90311737344,"internally_reserved":0,"allocated":4155187200,"data_stored":9144665716,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1097352,"internal_metadata":2168734072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,6,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.84499999999999997,"15min":0.84699999999999998},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":7.7140000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95599999999999996,"15min":0.90600000000000003},"min":{"1min":0.27200000000000002,"5min":0.247,"15min":0.21299999999999999},"max":{"1min":4.3540000000000001,"5min":4.3540000000000001,"15min":7.8559999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:09:07 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.98399999999999999,"15min":0.98699999999999999},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":4.907,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.0049999999999999,"15min":1.0720000000000001},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.929,"5min":4.8090000000000002,"15min":8.4000000000000004},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4154204160,"data_stored":9144250368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4001488896,"data_stored":8813350931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8155942912,"data_stored":17957851155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:07.357 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:07.357 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:07.357 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:07.358 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:07.358 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:10:10.516 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:10:10.515+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:10:11.565 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:10:11.561+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:10:11.565 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:10:11.561+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:10:27.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:27.902 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:10:27.903 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:10:27.922 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2260,"stamp":"2023-10-21T05:10:27.807432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29156538109,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2050575,"num_read_kb":7936102,"num_write":2001090,"num_write_kb":8002135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84328,"ondisk_log_size":84328,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21303152,"kb_used_data":15997976,"kb_used_omap":4159,"kb_used_meta":5300992,"kb_avail":261812368,"statfs":{"total":289910292480,"available":268095864832,"internally_reserved":0,"allocated":16381927424,"data_stored":35983503834,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4259014,"internal_metadata":5428216634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,3,7,10,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20365312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4733,"num_read_kb":18872,"num_write":4934,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"2.7","version":"138'6694","reported_seq":9948,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:25.068436+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:10:25.068436+0000","last_peered":"2023-10-21T05:10:25.068436+0000","last_clean":"2023-10-21T05:10:25.068436+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:10:25.068436+0000","last_undegraded":"2023-10-21T05:10:25.068436+0000","last_fullsized":"2023-10-21T05:10:25.068436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'3430","reported_seq":7091,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:24.749335+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:10:24.749335+0000","last_peered":"2023-10-21T05:10:24.749335+0000","last_clean":"2023-10-21T05:10:24.749335+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:10:24.749335+0000","last_undegraded":"2023-10-21T05:10:24.749335+0000","last_fullsized":"2023-10-21T05:10:24.749335+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'9990","reported_seq":14737,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:25.081730+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:10:25.081730+0000","last_peered":"2023-10-21T05:10:25.081730+0000","last_clean":"2023-10-21T05:10:25.081730+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:10:25.081730+0000","last_undegraded":"2023-10-21T05:10:25.081730+0000","last_fullsized":"2023-10-21T05:10:25.081730+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'6734","reported_seq":11002,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:25.060139+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:10:25.060139+0000","last_peered":"2023-10-21T05:10:25.060139+0000","last_clean":"2023-10-21T05:10:25.060139+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:10:25.060139+0000","last_undegraded":"2023-10-21T05:10:25.060139+0000","last_fullsized":"2023-10-21T05:10:25.060139+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6734,"log_dups_size":0,"ondisk_log_size":6734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'501859","reported_seq":992755,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:25.495963+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:10:25.495963+0000","last_peered":"2023-10-21T05:10:25.495963+0000","last_clean":"2023-10-21T05:10:25.495963+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:10:25.495963+0000","last_undegraded":"2023-10-21T05:10:25.495963+0000","last_fullsized":"2023-10-21T05:10:25.495963+0000","mapping_epoch":14,"log_start":"137'491857","ondisk_log_start":"137'491857","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2337,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 247s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7210668032,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":495526,"num_read_kb":1955012,"num_write":494933,"num_write_kb":1979732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'13626","reported_seq":20103,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:24.273855+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:10:24.273855+0000","last_peered":"2023-10-21T05:10:24.273855+0000","last_clean":"2023-10-21T05:10:24.273855+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:10:24.273855+0000","last_undegraded":"2023-10-21T05:10:24.273855+0000","last_fullsized":"2023-10-21T05:10:24.273855+0000","mapping_epoch":11,"log_start":"72'3600","ondisk_log_start":"72'3600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'507275","reported_seq":1004928,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:25.986402+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:10:25.986402+0000","last_peered":"2023-10-21T05:10:25.986402+0000","last_clean":"2023-10-21T05:10:25.986402+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:10:25.986402+0000","last_undegraded":"2023-10-21T05:10:25.986402+0000","last_fullsized":"2023-10-21T05:10:25.986402+0000","mapping_epoch":14,"log_start":"137'497258","ondisk_log_start":"137'497258","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1938,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 231s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7299985408,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":502344,"num_read_kb":1981336,"num_write":500638,"num_write_kb":2002552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3388,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:22.339662+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:10:22.339662+0000","last_peered":"2023-10-21T05:10:22.339662+0000","last_clean":"2023-10-21T05:10:22.339662+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:10:22.339662+0000","last_undegraded":"2023-10-21T05:10:22.339662+0000","last_fullsized":"2023-10-21T05:10:22.339662+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'530109","reported_seq":1046300,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:25.469953+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:10:25.469953+0000","last_peered":"2023-10-21T05:10:25.469953+0000","last_clean":"2023-10-21T05:10:25.469953+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:10:25.469953+0000","last_undegraded":"2023-10-21T05:10:25.469953+0000","last_fullsized":"2023-10-21T05:10:25.469953+0000","mapping_epoch":14,"log_start":"137'520048","ondisk_log_start":"137'520048","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1882,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 244s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7625080832,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":521126,"num_read_kb":2056272,"num_write":523279,"num_write_kb":2093116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6750,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:22.427484+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:10:22.427484+0000","last_peered":"2023-10-21T05:10:22.427484+0000","last_clean":"2023-10-21T05:10:22.427484+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:10:22.427484+0000","last_undegraded":"2023-10-21T05:10:22.427484+0000","last_fullsized":"2023-10-21T05:10:22.427484+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'485951","reported_seq":963482,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:25.996016+0000","last_change":"2023-10-21T05:10:23.125680+0000","last_active":"2023-10-21T05:10:25.996016+0000","last_peered":"2023-10-21T05:10:25.996016+0000","last_clean":"2023-10-21T05:10:25.996016+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:10:25.996016+0000","last_undegraded":"2023-10-21T05:10:25.996016+0000","last_fullsized":"2023-10-21T05:10:25.996016+0000","mapping_epoch":14,"log_start":"136'475856","ondisk_log_start":"136'475856","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7020208147,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":482120,"num_read_kb":1902412,"num_write":481082,"num_write_kb":1924322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"138'6657","reported_seq":9842,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:25.074470+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:10:25.074470+0000","last_peered":"2023-10-21T05:10:25.074470+0000","last_clean":"2023-10-21T05:10:25.074470+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:10:25.074470+0000","last_undegraded":"2023-10-21T05:10:25.074470+0000","last_fullsized":"2023-10-21T05:10:25.074470+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6657,"log_dups_size":0,"ondisk_log_size":6657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":311,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:09:59.399743+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:09:59.399743+0000","last_peered":"2023-10-21T05:09:59.399743+0000","last_clean":"2023-10-21T05:09:59.399743+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:09:59.399743+0000","last_undegraded":"2023-10-21T05:09:59.399743+0000","last_fullsized":"2023-10-21T05:09:59.399743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29155942419,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2001116,"num_read_kb":7895032,"num_write":1999932,"num_write_kb":7999722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16379617280,"data_stored":35983433766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40175,"ondisk_log_size":40175,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44054,"ondisk_log_size":44054,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739220,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10072556,"kb_used_data":7996068,"kb_used_omap":2048,"kb_used_meta":2074431,"kb_avail":84299284,"statfs":{"total":96636764160,"available":86322466816,"internally_reserved":0,"allocated":8187973632,"data_stored":17988939209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2097989,"internal_metadata":2124217531},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.014,"15min":0.98899999999999999},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":2.9430000000000001,"5min":3.8170000000000002,"15min":6.5419999999999998},"last":0.318},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95799999999999996,"15min":0.93799999999999994},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.8500000000000001,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.36599999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.1020000000000001,"15min":1.1699999999999999},"min":{"1min":0.33000000000000002,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":3.0150000000000001,"5min":5.3319999999999999,"15min":27.196999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.268,"15min":1.262},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":2.907,"5min":10.253,"15min":27.242999999999999},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739220,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5939624,"kb_used_data":3928096,"kb_used_omap":1037,"kb_used_meta":2010482,"kb_avail":88432216,"statfs":{"total":96636764160,"available":90554589184,"internally_reserved":0,"allocated":4022370304,"data_stored":8833502621,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1061993,"internal_metadata":2058734487},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.90200000000000002,"15min":0.90800000000000003},"min":{"1min":0.218,"5min":0.186,"15min":0.16400000000000001},"max":{"1min":4.8929999999999998,"5min":10.711,"15min":10.711},"last":0.248},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.054,"15min":0.98099999999999998},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":2.5449999999999999,"5min":10.512,"15min":10.512},"last":0.28299999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:09:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.05,"15min":1.0700000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":3,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.004,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.30299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":2.8490000000000002,"5min":4.8650000000000002,"15min":6},"last":0.47899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5290972,"kb_used_data":4073812,"kb_used_omap":1073,"kb_used_meta":1216078,"kb_avail":89080868,"statfs":{"total":96636764160,"available":91218808832,"internally_reserved":0,"allocated":4171583488,"data_stored":9161062004,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1245264616},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.84499999999999997,"15min":0.84699999999999998},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":7.7140000000000004},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95599999999999996,"15min":0.90600000000000003},"min":{"1min":0.27200000000000002,"5min":0.247,"15min":0.21299999999999999},"max":{"1min":4.3540000000000001,"5min":4.3540000000000001,"15min":7.8559999999999999},"last":1.0249999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:10:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":0.98399999999999999},"min":{"1min":0.21299999999999999,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":3.677,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.0680000000000001},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.133,"5min":5.133,"15min":7.8159999999999998},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4170600448,"data_stored":9160646656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4021125120,"data_stored":8832987155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8187891712,"data_stored":17989799955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:27.924 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:27.924 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:27.925 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:27.925 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:27.926 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:10:47.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:48.469 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:10:48.469 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:10:48.489 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2023-10-21T05:10:47.810952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29188163325,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2058516,"num_read_kb":7967794,"num_write":2008813,"num_write_kb":8033027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84538,"ondisk_log_size":84538,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20516792,"kb_used_data":16059360,"kb_used_omap":4380,"kb_used_meta":4453027,"kb_avail":262598728,"statfs":{"total":289910292480,"available":268901097472,"internally_reserved":0,"allocated":16444784640,"data_stored":36046361050,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":4559900636},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,8,5,32,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19722240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4904,"num_read_kb":19572,"num_write":4817,"num_write_kb":19268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"2.7","version":"138'6730","reported_seq":9992,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.719818+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:10:44.719818+0000","last_peered":"2023-10-21T05:10:44.719818+0000","last_clean":"2023-10-21T05:10:44.719818+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:10:44.719818+0000","last_undegraded":"2023-10-21T05:10:44.719818+0000","last_fullsized":"2023-10-21T05:10:44.719818+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'3448","reported_seq":7114,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.166139+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:10:44.166139+0000","last_peered":"2023-10-21T05:10:44.166139+0000","last_clean":"2023-10-21T05:10:44.166139+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:10:44.166139+0000","last_undegraded":"2023-10-21T05:10:44.166139+0000","last_fullsized":"2023-10-21T05:10:44.166139+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'10044","reported_seq":14803,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.735372+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:10:44.735372+0000","last_peered":"2023-10-21T05:10:44.735372+0000","last_clean":"2023-10-21T05:10:44.735372+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:10:44.735372+0000","last_undegraded":"2023-10-21T05:10:44.735372+0000","last_fullsized":"2023-10-21T05:10:44.735372+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'6770","reported_seq":11046,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.720104+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:10:44.720104+0000","last_peered":"2023-10-21T05:10:44.720104+0000","last_clean":"2023-10-21T05:10:44.720104+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:10:44.720104+0000","last_undegraded":"2023-10-21T05:10:44.720104+0000","last_fullsized":"2023-10-21T05:10:44.720104+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'503903","reported_seq":996758,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:45.591045+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:10:45.591045+0000","last_peered":"2023-10-21T05:10:45.591045+0000","last_clean":"2023-10-21T05:10:45.591045+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:10:45.591045+0000","last_undegraded":"2023-10-21T05:10:45.591045+0000","last_fullsized":"2023-10-21T05:10:45.591045+0000","mapping_epoch":14,"log_start":"137'493857","ondisk_log_start":"137'493857","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2499,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 267s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7218495488,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":497467,"num_read_kb":1962756,"num_write":496846,"num_write_kb":1987384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'13666","reported_seq":20159,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.366950+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:10:44.366950+0000","last_peered":"2023-10-21T05:10:44.366950+0000","last_clean":"2023-10-21T05:10:44.366950+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:10:44.366950+0000","last_undegraded":"2023-10-21T05:10:44.366950+0000","last_fullsized":"2023-10-21T05:10:44.366950+0000","mapping_epoch":11,"log_start":"72'3600","ondisk_log_start":"72'3600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'509362","reported_seq":1009001,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:46.167695+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:10:46.167695+0000","last_peered":"2023-10-21T05:10:46.167695+0000","last_clean":"2023-10-21T05:10:46.167695+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:10:46.167695+0000","last_undegraded":"2023-10-21T05:10:46.167695+0000","last_fullsized":"2023-10-21T05:10:46.167695+0000","mapping_epoch":14,"log_start":"137'499358","ondisk_log_start":"137'499358","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2088,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 251s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7308038144,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":504317,"num_read_kb":1989220,"num_write":502604,"num_write_kb":2010416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3396,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:42.817269+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:10:42.817269+0000","last_peered":"2023-10-21T05:10:42.817269+0000","last_clean":"2023-10-21T05:10:42.817269+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:10:42.817269+0000","last_undegraded":"2023-10-21T05:10:42.817269+0000","last_fullsized":"2023-10-21T05:10:42.817269+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'532194","reported_seq":1050456,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:45.585358+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:10:45.585358+0000","last_peered":"2023-10-21T05:10:45.585358+0000","last_clean":"2023-10-21T05:10:45.585358+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:10:45.585358+0000","last_undegraded":"2023-10-21T05:10:45.585358+0000","last_fullsized":"2023-10-21T05:10:45.585358+0000","mapping_epoch":14,"log_start":"137'522148","ondisk_log_start":"137'522148","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2024,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 264s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7633158144,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":523183,"num_read_kb":2064480,"num_write":525249,"num_write_kb":2100996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6766,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:42.822803+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:10:42.822803+0000","last_peered":"2023-10-21T05:10:42.822803+0000","last_clean":"2023-10-21T05:10:42.822803+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:10:42.822803+0000","last_undegraded":"2023-10-21T05:10:42.822803+0000","last_fullsized":"2023-10-21T05:10:42.822803+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'487825","reported_seq":967339,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.166655+0000","last_change":"2023-10-21T05:10:45.247634+0000","last_active":"2023-10-21T05:10:46.166655+0000","last_peered":"2023-10-21T05:10:46.166655+0000","last_clean":"2023-10-21T05:10:46.166655+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:10:46.166655+0000","last_undegraded":"2023-10-21T05:10:46.166655+0000","last_fullsized":"2023-10-21T05:10:46.166655+0000","mapping_epoch":14,"log_start":"137'477756","ondisk_log_start":"137'477756","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7027875859,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":484090,"num_read_kb":1910268,"num_write":482956,"num_write_kb":1931818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"138'6693","reported_seq":9886,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.748314+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:10:44.748314+0000","last_peered":"2023-10-21T05:10:44.748314+0000","last_clean":"2023-10-21T05:10:44.748314+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:10:44.748314+0000","last_undegraded":"2023-10-21T05:10:44.748314+0000","last_fullsized":"2023-10-21T05:10:44.748314+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6693,"log_dups_size":0,"ondisk_log_size":6693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":311,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:09:59.399743+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:09:59.399743+0000","last_peered":"2023-10-21T05:09:59.399743+0000","last_clean":"2023-10-21T05:09:59.399743+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:09:59.399743+0000","last_undegraded":"2023-10-21T05:09:59.399743+0000","last_fullsized":"2023-10-21T05:09:59.399743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29187567635,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2009057,"num_read_kb":7926724,"num_write":2007655,"num_write_kb":8030614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16442671104,"data_stored":36046487590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44274,"ondisk_log_size":44274,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739224,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10099332,"kb_used_data":8026620,"kb_used_omap":2268,"kb_used_meta":2070435,"kb_avail":84272508,"statfs":{"total":96636764160,"available":86295048192,"internally_reserved":0,"allocated":8219258880,"data_stored":18020224457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":2120126444},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.014,"15min":0.98899999999999999},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":2.9430000000000001,"5min":3.8170000000000002,"15min":6.5419999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95799999999999996,"15min":0.93799999999999994},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.8500000000000001,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.1020000000000001,"15min":1.1699999999999999},"min":{"1min":0.33000000000000002,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":3.0150000000000001,"5min":5.3319999999999999,"15min":27.196999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.268,"15min":1.262},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":2.907,"5min":10.253,"15min":27.242999999999999},"last":0.49299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739224,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5034484,"kb_used_data":3943276,"kb_used_omap":1038,"kb_used_meta":1090161,"kb_avail":89337356,"statfs":{"total":96636764160,"available":91481452544,"internally_reserved":0,"allocated":4037914624,"data_stored":8849046941,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1116325128},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,24,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.90200000000000002,"15min":0.90800000000000003},"min":{"1min":0.218,"5min":0.186,"15min":0.16400000000000001},"max":{"1min":4.8929999999999998,"5min":10.711,"15min":10.711},"last":6.0899999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.054,"15min":0.98099999999999998},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":2.5449999999999999,"5min":10.512,"15min":10.512},"last":6.0579999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:09:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.05,"15min":1.0700000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":3,"5min":10.167999999999999,"15min":10.167999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.30299999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":2.8490000000000002,"5min":4.8650000000000002,"15min":6},"last":6.1230000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5382976,"kb_used_data":4089464,"kb_used_omap":1073,"kb_used_meta":1292430,"kb_avail":88988864,"statfs":{"total":96636764160,"available":91124596736,"internally_reserved":0,"allocated":4187611136,"data_stored":9177089652,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1323449064},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,2,5,7,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.84499999999999997,"15min":0.84699999999999998},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":7.7140000000000004},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95599999999999996,"15min":0.90600000000000003},"min":{"1min":0.27200000000000002,"5min":0.247,"15min":0.21299999999999999},"max":{"1min":4.3540000000000001,"5min":4.3540000000000001,"15min":7.8559999999999999},"last":1.6100000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:10:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":0.98399999999999999},"min":{"1min":0.21299999999999999,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":3.677,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.0680000000000001},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.133,"5min":5.133,"15min":7.8159999999999998},"last":2.3140000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4186505216,"data_stored":9176551424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4036853760,"data_stored":8848715795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8219312128,"data_stored":18021220371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:48.490 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:48.491 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:48.491 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:48.492 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:48.492 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:05.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:11:05.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:11:05.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:11:05.452 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:11:05.452 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:11:05.452 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:11:05.453 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:11:05.453 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:11:05.453 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:05.453 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:05.448+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:11:08.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:11:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:11:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2280,"stamp":"2023-10-21T05:11:07.814484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29219829501,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2066277,"num_read_kb":7998762,"num_write":2016513,"num_write_kb":8063827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84658,"ondisk_log_size":84658,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19458904,"kb_used_data":16122368,"kb_used_omap":4380,"kb_used_meta":3332131,"kb_avail":263656616,"statfs":{"total":289910292480,"available":269984374784,"internally_reserved":0,"allocated":16509304832,"data_stored":36110881242,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":3412103132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,13,9,9,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19763200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4790,"num_read_kb":19116,"num_write":4810,"num_write_kb":19240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002161"},"pg_stats":[{"pgid":"2.7","version":"138'6750","reported_seq":10020,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:05.459559+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:11:05.459559+0000","last_peered":"2023-10-21T05:11:05.459559+0000","last_clean":"2023-10-21T05:11:05.459559+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:11:05.459559+0000","last_undegraded":"2023-10-21T05:11:05.459559+0000","last_fullsized":"2023-10-21T05:11:05.459559+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6750,"log_dups_size":0,"ondisk_log_size":6750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'3458","reported_seq":7128,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:04.488132+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:11:04.488132+0000","last_peered":"2023-10-21T05:11:04.488132+0000","last_clean":"2023-10-21T05:11:04.488132+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:11:04.488132+0000","last_undegraded":"2023-10-21T05:11:04.488132+0000","last_fullsized":"2023-10-21T05:11:04.488132+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'10074","reported_seq":14845,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:05.458668+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:11:05.458668+0000","last_peered":"2023-10-21T05:11:05.458668+0000","last_clean":"2023-10-21T05:11:05.458668+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:11:05.458668+0000","last_undegraded":"2023-10-21T05:11:05.458668+0000","last_fullsized":"2023-10-21T05:11:05.458668+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'6790","reported_seq":11074,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:05.446085+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:11:05.446085+0000","last_peered":"2023-10-21T05:11:05.446085+0000","last_clean":"2023-10-21T05:11:05.446085+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:11:05.446085+0000","last_undegraded":"2023-10-21T05:11:05.446085+0000","last_fullsized":"2023-10-21T05:11:05.446085+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'505948","reported_seq":1000736,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:05.684346+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:11:05.684346+0000","last_peered":"2023-10-21T05:11:05.684346+0000","last_clean":"2023-10-21T05:11:05.684346+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:11:05.684346+0000","last_undegraded":"2023-10-21T05:11:05.684346+0000","last_fullsized":"2023-10-21T05:11:05.684346+0000","mapping_epoch":14,"log_start":"137'495857","ondisk_log_start":"137'495857","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2629,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 287s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7226548224,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":499385,"num_read_kb":1970396,"num_write":498786,"num_write_kb":1995144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'13706","reported_seq":20215,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.297101+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:11:03.297101+0000","last_peered":"2023-10-21T05:11:03.297101+0000","last_clean":"2023-10-21T05:11:03.297101+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:11:03.297101+0000","last_undegraded":"2023-10-21T05:11:03.297101+0000","last_fullsized":"2023-10-21T05:11:03.297101+0000","mapping_epoch":11,"log_start":"74'3700","ondisk_log_start":"74'3700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'511431","reported_seq":1013023,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:06.273575+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:11:06.273575+0000","last_peered":"2023-10-21T05:11:06.273575+0000","last_clean":"2023-10-21T05:11:06.273575+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:11:06.273575+0000","last_undegraded":"2023-10-21T05:11:06.273575+0000","last_fullsized":"2023-10-21T05:11:06.273575+0000","mapping_epoch":14,"log_start":"137'501358","ondisk_log_start":"137'501358","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2272,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 271s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7315902464,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":506258,"num_read_kb":1996972,"num_write":504523,"num_write_kb":2018092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3405,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.299566+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:11:03.299566+0000","last_peered":"2023-10-21T05:11:03.299566+0000","last_clean":"2023-10-21T05:11:03.299566+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:11:03.299566+0000","last_undegraded":"2023-10-21T05:11:03.299566+0000","last_fullsized":"2023-10-21T05:11:03.299566+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'534311","reported_seq":1054615,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:05.693328+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:11:05.693328+0000","last_peered":"2023-10-21T05:11:05.693328+0000","last_clean":"2023-10-21T05:11:05.693328+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:11:05.693328+0000","last_undegraded":"2023-10-21T05:11:05.693328+0000","last_fullsized":"2023-10-21T05:11:05.693328+0000","mapping_epoch":14,"log_start":"137'524248","ondisk_log_start":"137'524248","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2179,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 284s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7641321472,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":525212,"num_read_kb":2072576,"num_write":527241,"num_write_kb":2108964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6782,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.341980+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:11:03.341980+0000","last_peered":"2023-10-21T05:11:03.341980+0000","last_clean":"2023-10-21T05:11:03.341980+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:11:03.341980+0000","last_undegraded":"2023-10-21T05:11:03.341980+0000","last_fullsized":"2023-10-21T05:11:03.341980+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'489674","reported_seq":971063,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:06.270984+0000","last_change":"2023-10-21T05:11:05.404153+0000","last_active":"2023-10-21T05:11:06.270984+0000","last_peered":"2023-10-21T05:11:06.270984+0000","last_clean":"2023-10-21T05:11:06.270984+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:11:06.270984+0000","last_undegraded":"2023-10-21T05:11:06.270984+0000","last_fullsized":"2023-10-21T05:11:06.270984+0000","mapping_epoch":14,"log_start":"137'479656","ondisk_log_start":"137'479656","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7035461651,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":485963,"num_read_kb":1917748,"num_write":484805,"num_write_kb":1939214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"138'6713","reported_seq":9915,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:11:05.472890+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:11:05.472890+0000","last_peered":"2023-10-21T05:11:05.472890+0000","last_clean":"2023-10-21T05:11:05.472890+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:11:05.472890+0000","last_undegraded":"2023-10-21T05:11:05.472890+0000","last_fullsized":"2023-10-21T05:11:05.472890+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":311,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:09:59.399743+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:09:59.399743+0000","last_peered":"2023-10-21T05:09:59.399743+0000","last_clean":"2023-10-21T05:09:59.399743+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:09:59.399743+0000","last_undegraded":"2023-10-21T05:09:59.399743+0000","last_fullsized":"2023-10-21T05:09:59.399743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29219233811,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2016818,"num_read_kb":7957692,"num_write":2015355,"num_write_kb":8061414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16505692160,"data_stored":36109508646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40245,"ondisk_log_size":40245,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44314,"ondisk_log_size":44314,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8893304,"kb_used_data":8058672,"kb_used_omap":2268,"kb_used_meta":832355,"kb_avail":85478536,"statfs":{"total":96636764160,"available":87530020864,"internally_reserved":0,"allocated":8252080128,"data_stored":18053045705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":852332524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.014,"15min":0.98899999999999999},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":2.9430000000000001,"5min":3.8170000000000002,"15min":6.5419999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95799999999999996,"15min":0.93799999999999994},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.8500000000000001,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.307}]},{"osd":1,"last update":"Sat Oct 21 05:10:04 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.1020000000000001,"15min":1.1699999999999999},"min":{"1min":0.33000000000000002,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":3.0150000000000001,"5min":5.3319999999999999,"15min":27.196999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.268,"15min":1.262},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":2.907,"5min":10.253,"15min":27.242999999999999},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739228,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5129364,"kb_used_data":3958476,"kb_used_omap":1038,"kb_used_meta":1169841,"kb_avail":89242476,"statfs":{"total":96636764160,"available":91384295424,"internally_reserved":0,"allocated":4053479424,"data_stored":8864611741,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1197917448},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,1,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.79700000000000004,"15min":0.89700000000000002},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":6.0899999999999999,"5min":6.2140000000000004,"15min":10.711},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.96299999999999997,"15min":0.99399999999999999},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":6.0579999999999998,"5min":6.0579999999999998,"15min":10.512},"last":0.70399999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.92800000000000005,"15min":1.0580000000000001},"min":{"1min":0.26500000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":4.2640000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":0.31},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.0309999999999999,"15min":1.1619999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23100000000000001},"max":{"1min":6.1230000000000002,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.65600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739228,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5436236,"kb_used_data":4105220,"kb_used_omap":1073,"kb_used_meta":1329934,"kb_avail":88935604,"statfs":{"total":96636764160,"available":91070058496,"internally_reserved":0,"allocated":4203745280,"data_stored":9193223796,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1361853160},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.82899999999999996,"15min":0.85599999999999998},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.183},"max":{"1min":1.6779999999999999,"5min":4.3170000000000002,"15min":7.7140000000000004},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.93300000000000005},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":2.7919999999999998,"5min":4.3540000000000001,"15min":7.8559999999999999},"last":2.7919999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:10:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":0.98399999999999999},"min":{"1min":0.21299999999999999,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":3.677,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":2.8239999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.0680000000000001},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.133,"5min":5.133,"15min":7.8159999999999998},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4202618880,"data_stored":9192665088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4052291584,"data_stored":8864153619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8250781696,"data_stored":18052689939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:09.036 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:09.036 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:09.037 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:09.037 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:09.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:11:09.585 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:11:09.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:11:10.141 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:11:10.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:11:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:11:10.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:11:11.266 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:11:11.289 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:11.370 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:11:11.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:11:11.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:11:11.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:11:11.371 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:11:11.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:11:11.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:11:11.372 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:11:11.373 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:11.374 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:11.367+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:11:15.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:15.381+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:15.389 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:15.385+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:11:31.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:31.311 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:31.310+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:11:31.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:31.318+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:31.869 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:11:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:11:31.893 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2293,"stamp":"2023-10-21T05:11:31.818736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29259597565,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2076082,"num_read_kb":8037886,"num_write":2026205,"num_write_kb":8102595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84739,"ondisk_log_size":84739,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19482168,"kb_used_data":16200284,"kb_used_omap":4380,"kb_used_meta":3277475,"kb_avail":263633352,"statfs":{"total":289910292480,"available":269960552448,"internally_reserved":0,"allocated":16589090816,"data_stored":36190644702,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":3356135388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,8,8,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24281088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6043,"num_read_kb":24100,"num_write":5919,"num_write_kb":23676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002112"},"pg_stats":[{"pgid":"2.7","version":"139'6783","reported_seq":10064,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.330752+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:11:31.330752+0000","last_peered":"2023-10-21T05:11:31.330752+0000","last_clean":"2023-10-21T05:11:31.330752+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:11:31.330752+0000","last_undegraded":"2023-10-21T05:11:31.330752+0000","last_fullsized":"2023-10-21T05:11:31.330752+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6783,"log_dups_size":0,"ondisk_log_size":6783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'3475","reported_seq":7151,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.321028+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:11:31.321028+0000","last_peered":"2023-10-21T05:11:31.321028+0000","last_clean":"2023-10-21T05:11:31.321028+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:11:31.321028+0000","last_undegraded":"2023-10-21T05:11:31.321028+0000","last_fullsized":"2023-10-21T05:11:31.321028+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'10122","reported_seq":14910,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.338654+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:11:31.338654+0000","last_peered":"2023-10-21T05:11:31.338654+0000","last_clean":"2023-10-21T05:11:31.338654+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:11:31.338654+0000","last_undegraded":"2023-10-21T05:11:31.338654+0000","last_fullsized":"2023-10-21T05:11:31.338654+0000","mapping_epoch":11,"log_start":"38'100","ondisk_log_start":"38'100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'6822","reported_seq":11118,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.322223+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:11:31.322223+0000","last_peered":"2023-10-21T05:11:31.322223+0000","last_clean":"2023-10-21T05:11:31.322223+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:11:31.322223+0000","last_undegraded":"2023-10-21T05:11:31.322223+0000","last_fullsized":"2023-10-21T05:11:31.322223+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'508498","reported_seq":1005688,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:30.910804+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:11:30.910804+0000","last_peered":"2023-10-21T05:11:30.910804+0000","last_clean":"2023-10-21T05:11:30.910804+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:11:30.910804+0000","last_undegraded":"2023-10-21T05:11:30.910804+0000","last_fullsized":"2023-10-21T05:11:30.910804+0000","mapping_epoch":14,"log_start":"137'498458","ondisk_log_start":"137'498458","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2822,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 312s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7236354048,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":501770,"num_read_kb":1979904,"num_write":501178,"num_write_kb":2004712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"139'13770","reported_seq":20301,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:30.513057+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:11:30.513057+0000","last_peered":"2023-10-21T05:11:30.513057+0000","last_clean":"2023-10-21T05:11:30.513057+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:11:30.513057+0000","last_undegraded":"2023-10-21T05:11:30.513057+0000","last_fullsized":"2023-10-21T05:11:30.513057+0000","mapping_epoch":11,"log_start":"74'3700","ondisk_log_start":"74'3700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'514041","reported_seq":1018137,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:31.575913+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:11:31.575913+0000","last_peered":"2023-10-21T05:11:31.575913+0000","last_clean":"2023-10-21T05:11:31.575913+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:11:31.575913+0000","last_undegraded":"2023-10-21T05:11:31.575913+0000","last_fullsized":"2023-10-21T05:11:31.575913+0000","mapping_epoch":14,"log_start":"137'503958","ondisk_log_start":"137'503958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2479,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 296s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7325925376,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":508742,"num_read_kb":2006888,"num_write":506966,"num_write_kb":2027864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3414,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:23.784564+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:11:23.784564+0000","last_peered":"2023-10-21T05:11:23.784564+0000","last_clean":"2023-10-21T05:11:23.784564+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:11:23.784564+0000","last_undegraded":"2023-10-21T05:11:23.784564+0000","last_fullsized":"2023-10-21T05:11:23.784564+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'537019","reported_seq":1059928,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:30.894422+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:11:30.894422+0000","last_peered":"2023-10-21T05:11:30.894422+0000","last_clean":"2023-10-21T05:11:30.894422+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:11:30.894422+0000","last_undegraded":"2023-10-21T05:11:30.894422+0000","last_fullsized":"2023-10-21T05:11:30.894422+0000","mapping_epoch":14,"log_start":"137'526948","ondisk_log_start":"137'526948","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2348,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 309s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7651868672,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":527801,"num_read_kb":2082912,"num_write":529810,"num_write_kb":2119240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6799,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:23.793247+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:11:23.793247+0000","last_peered":"2023-10-21T05:11:23.793247+0000","last_clean":"2023-10-21T05:11:23.793247+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:11:23.793247+0000","last_undegraded":"2023-10-21T05:11:23.793247+0000","last_fullsized":"2023-10-21T05:11:23.793247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'491962","reported_seq":975705,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.583995+0000","last_change":"2023-10-21T05:11:30.551463+0000","last_active":"2023-10-21T05:11:31.583995+0000","last_peered":"2023-10-21T05:11:31.583995+0000","last_clean":"2023-10-21T05:11:31.583995+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:11:31.583995+0000","last_undegraded":"2023-10-21T05:11:31.583995+0000","last_fullsized":"2023-10-21T05:11:31.583995+0000","mapping_epoch":14,"log_start":"137'481956","ondisk_log_start":"137'481956","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7044853779,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":488310,"num_read_kb":1927112,"num_write":487093,"num_write_kb":1948366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"139'6745","reported_seq":9958,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.367679+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:11:31.367679+0000","last_peered":"2023-10-21T05:11:31.367679+0000","last_clean":"2023-10-21T05:11:31.367679+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:11:31.367679+0000","last_undegraded":"2023-10-21T05:11:31.367679+0000","last_fullsized":"2023-10-21T05:11:31.367679+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":312,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:20.938843+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:10:20.938843+0000","last_peered":"2023-10-21T05:10:20.938843+0000","last_clean":"2023-10-21T05:10:20.938843+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:10:20.938843+0000","last_undegraded":"2023-10-21T05:10:20.938843+0000","last_fullsized":"2023-10-21T05:10:20.938843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29259001875,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2026623,"num_read_kb":7996816,"num_write":2025047,"num_write_kb":8100182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16585191424,"data_stored":36189007910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44440,"ondisk_log_size":44440,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8723144,"kb_used_data":8097280,"kb_used_omap":2268,"kb_used_meta":623587,"kb_avail":85648696,"statfs":{"total":96636764160,"available":87704264704,"internally_reserved":0,"allocated":8291614720,"data_stored":18092572789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":638554092},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:05 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.85099999999999998,"15min":0.93799999999999994},"min":{"1min":0.21199999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":1.476,"5min":3.8170000000000002,"15min":6.5419999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.88700000000000001,"15min":0.92600000000000005},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.883,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Oct 21 05:11:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.99199999999999999,"15min":1.1319999999999999},"min":{"1min":0.26300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.024,"5min":5.3319999999999999,"15min":27.196999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0780000000000001,"15min":1.23},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.224},"max":{"1min":2.2549999999999999,"5min":10.253,"15min":27.242999999999999},"last":0.91500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739233,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208620,"kb_used_data":3978016,"kb_used_omap":1038,"kb_used_meta":1229553,"kb_avail":89163220,"statfs":{"total":96636764160,"available":91303137280,"internally_reserved":0,"allocated":4073488384,"data_stored":8884613193,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1259062536},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.79700000000000004,"15min":0.89700000000000002},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":6.0899999999999999,"5min":6.2140000000000004,"15min":10.711},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.96299999999999997,"15min":0.99399999999999999},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":6.0579999999999998,"5min":6.0579999999999998,"15min":10.512},"last":0.78400000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.92800000000000005,"15min":1.0580000000000001},"min":{"1min":0.26500000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":4.2640000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":2.8700000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.0309999999999999,"15min":1.1619999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23100000000000001},"max":{"1min":6.1230000000000002,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.72499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5550404,"kb_used_data":4124988,"kb_used_omap":1073,"kb_used_meta":1424334,"kb_avail":88821436,"statfs":{"total":96636764160,"available":90953150464,"internally_reserved":0,"allocated":4223987712,"data_stored":9213458720,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1458518760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.82899999999999996,"15min":0.85599999999999998},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.183},"max":{"1min":1.6779999999999999,"5min":4.3170000000000002,"15min":7.7140000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.93300000000000005},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":2.7919999999999998,"5min":4.3540000000000001,"15min":7.8559999999999999},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.022,"15min":0.97599999999999998},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":3.5,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":1.8089999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0029999999999999,"15min":1.0620000000000001},"min":{"1min":0.32300000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":3.4540000000000002,"5min":5.133,"15min":7.8159999999999998},"last":2.4940000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4222926848,"data_stored":9212973056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4071661568,"data_stored":8883523603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8290603008,"data_stored":18092511251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:31.894 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:31.895 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:31.895 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:31.896 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:31.896 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:46.616 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.614+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:11:46.621 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.618+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:11:46.625 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.623+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:11:46.626 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.623+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:11:46.631 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.629+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:11:46.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.629+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:11:46.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.629+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:11:46.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.629+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:46.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:46.629+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:11:51.118 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.113+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:11:51.119 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.113+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:11:51.119 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.115+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:11:51.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.118+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:11:51.121 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.118+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:11:51.128 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.126+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:11:51.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.126+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:11:51.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.126+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:11:51.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.126+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:51.130 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:51.126+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:11:51.897 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:52.455 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:11:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:11:52.482 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2303,"stamp":"2023-10-21T05:11:51.822184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29286860541,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2082999,"num_read_kb":8065518,"num_write":2032848,"num_write_kb":8129167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84805,"ondisk_log_size":84805,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19797260,"kb_used_data":16256372,"kb_used_omap":4380,"kb_used_meta":3536483,"kb_avail":263318260,"statfs":{"total":289910292480,"available":269637898240,"internally_reserved":0,"allocated":16646524928,"data_stored":36248078814,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":3621359580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,6,17,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19550208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4938,"num_read_kb":19724,"num_write":4769,"num_write_kb":19076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002024"},"pg_stats":[{"pgid":"2.7","version":"139'6815","reported_seq":10103,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:46.636040+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:11:46.636040+0000","last_peered":"2023-10-21T05:11:46.636040+0000","last_clean":"2023-10-21T05:11:46.636040+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:11:46.636040+0000","last_undegraded":"2023-10-21T05:11:46.636040+0000","last_fullsized":"2023-10-21T05:11:46.636040+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'3491","reported_seq":7170,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:45.991997+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:11:45.991997+0000","last_peered":"2023-10-21T05:11:45.991997+0000","last_clean":"2023-10-21T05:11:45.991997+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:11:45.991997+0000","last_undegraded":"2023-10-21T05:11:45.991997+0000","last_fullsized":"2023-10-21T05:11:45.991997+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'10170","reported_seq":14967,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:46.636402+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:11:46.636402+0000","last_peered":"2023-10-21T05:11:46.636402+0000","last_clean":"2023-10-21T05:11:46.636402+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:11:46.636402+0000","last_undegraded":"2023-10-21T05:11:46.636402+0000","last_fullsized":"2023-10-21T05:11:46.636402+0000","mapping_epoch":11,"log_start":"38'100","ondisk_log_start":"38'100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'6854","reported_seq":11156,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:46.629352+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:11:46.629352+0000","last_peered":"2023-10-21T05:11:46.629352+0000","last_clean":"2023-10-21T05:11:46.629352+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:11:46.629352+0000","last_undegraded":"2023-10-21T05:11:46.629352+0000","last_fullsized":"2023-10-21T05:11:46.629352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'510501","reported_seq":1009621,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:51.134704+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:11:51.134704+0000","last_peered":"2023-10-21T05:11:51.134704+0000","last_clean":"2023-10-21T05:11:51.134704+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:11:51.134704+0000","last_undegraded":"2023-10-21T05:11:51.134704+0000","last_fullsized":"2023-10-21T05:11:51.134704+0000","mapping_epoch":14,"log_start":"138'500458","ondisk_log_start":"138'500458","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2992,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7243984896,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":503689,"num_read_kb":1987572,"num_write":503039,"num_write_kb":2012156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"139'13870","reported_seq":20417,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:51.009608+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:11:51.009608+0000","last_peered":"2023-10-21T05:11:51.009608+0000","last_clean":"2023-10-21T05:11:51.009608+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:11:51.009608+0000","last_undegraded":"2023-10-21T05:11:51.009608+0000","last_fullsized":"2023-10-21T05:11:51.009608+0000","mapping_epoch":11,"log_start":"74'3800","ondisk_log_start":"74'3800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'515593","reported_seq":1021180,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:46.844321+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:11:46.844321+0000","last_peered":"2023-10-21T05:11:46.844321+0000","last_clean":"2023-10-21T05:11:46.844321+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:11:46.844321+0000","last_undegraded":"2023-10-21T05:11:46.844321+0000","last_fullsized":"2023-10-21T05:11:46.844321+0000","mapping_epoch":14,"log_start":"138'505558","ondisk_log_start":"138'505558","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2568,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 311s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7331999744,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":510225,"num_read_kb":2012808,"num_write":508445,"num_write_kb":2033780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3422,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:44.261505+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:11:44.261505+0000","last_peered":"2023-10-21T05:11:44.261505+0000","last_clean":"2023-10-21T05:11:44.261505+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:11:44.261505+0000","last_undegraded":"2023-10-21T05:11:44.261505+0000","last_fullsized":"2023-10-21T05:11:44.261505+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'539073","reported_seq":1064024,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:51.091962+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:11:51.091962+0000","last_peered":"2023-10-21T05:11:51.091962+0000","last_clean":"2023-10-21T05:11:51.091962+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:11:51.091962+0000","last_undegraded":"2023-10-21T05:11:51.091962+0000","last_fullsized":"2023-10-21T05:11:51.091962+0000","mapping_epoch":14,"log_start":"138'529049","ondisk_log_start":"138'529049","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2529,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 330s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7659700224,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":529837,"num_read_kb":2091048,"num_write":531715,"num_write_kb":2126860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6815,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:44.258127+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:11:44.258127+0000","last_peered":"2023-10-21T05:11:44.258127+0000","last_clean":"2023-10-21T05:11:44.258127+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:11:44.258127+0000","last_undegraded":"2023-10-21T05:11:44.258127+0000","last_fullsized":"2023-10-21T05:11:44.258127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'493360","reported_seq":978587,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:46.843888+0000","last_change":"2023-10-21T05:11:44.600749+0000","last_active":"2023-10-21T05:11:46.843888+0000","last_peered":"2023-10-21T05:11:46.843888+0000","last_clean":"2023-10-21T05:11:46.843888+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:11:46.843888+0000","last_undegraded":"2023-10-21T05:11:46.843888+0000","last_fullsized":"2023-10-21T05:11:46.843888+0000","mapping_epoch":14,"log_start":"137'483356","ondisk_log_start":"137'483356","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7050579987,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":489789,"num_read_kb":1933020,"num_write":488491,"num_write_kb":1953958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"139'6777","reported_seq":9996,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:11:46.658048+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:11:46.658048+0000","last_peered":"2023-10-21T05:11:46.658048+0000","last_clean":"2023-10-21T05:11:46.658048+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:11:46.658048+0000","last_undegraded":"2023-10-21T05:11:46.658048+0000","last_fullsized":"2023-10-21T05:11:46.658048+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":312,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:20.938843+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:10:20.938843+0000","last_peered":"2023-10-21T05:10:20.938843+0000","last_clean":"2023-10-21T05:10:20.938843+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:10:20.938843+0000","last_undegraded":"2023-10-21T05:10:20.938843+0000","last_fullsized":"2023-10-21T05:10:20.938843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29286264851,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2033540,"num_read_kb":8024448,"num_write":2031690,"num_write_kb":8126754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16642928640,"data_stored":36246745126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40106,"ondisk_log_size":40106,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44600,"ondisk_log_size":44600,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739237,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8875128,"kb_used_data":8127088,"kb_used_omap":2268,"kb_used_meta":745763,"kb_avail":85496712,"statfs":{"total":96636764160,"available":87548633088,"internally_reserved":0,"allocated":8322138112,"data_stored":18123096181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":763662316},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:05 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.85099999999999998,"15min":0.93799999999999994},"min":{"1min":0.21199999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":1.476,"5min":3.8170000000000002,"15min":6.5419999999999998},"last":2.6659999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.88700000000000001,"15min":0.92600000000000005},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.20599999999999999},"max":{"1min":2.883,"5min":3.8159999999999998,"15min":7.2709999999999999},"last":4.4219999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:11:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.99199999999999999,"15min":1.1319999999999999},"min":{"1min":0.26300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.024,"5min":5.3319999999999999,"15min":27.196999999999999},"last":2.8490000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0780000000000001,"15min":1.23},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.224},"max":{"1min":2.2549999999999999,"5min":10.253,"15min":27.242999999999999},"last":2.887}]}]},{"osd":1,"up_from":8,"seq":34359739236,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5268536,"kb_used_data":3989360,"kb_used_omap":1038,"kb_used_meta":1278129,"kb_avail":89103304,"statfs":{"total":96636764160,"available":91241783296,"internally_reserved":0,"allocated":4085104640,"data_stored":8896229449,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1308804360},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.79700000000000004,"15min":0.89700000000000002},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":6.0899999999999999,"5min":6.2140000000000004,"15min":10.711},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.96299999999999997,"15min":0.99399999999999999},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":6.0579999999999998,"5min":6.0579999999999998,"15min":10.512},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.92800000000000005,"15min":1.0580000000000001},"min":{"1min":0.26500000000000001,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":4.2640000000000002,"5min":10.167999999999999,"15min":10.167999999999999},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.0309999999999999,"15min":1.1619999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23100000000000001},"max":{"1min":6.1230000000000002,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":1.361}]}]},{"osd":0,"up_from":8,"seq":34359739237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5653596,"kb_used_data":4139924,"kb_used_omap":1073,"kb_used_meta":1512590,"kb_avail":88718244,"statfs":{"total":96636764160,"available":90847481856,"internally_reserved":0,"allocated":4239282176,"data_stored":9228753184,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1548892904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,11,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.82899999999999996,"15min":0.85599999999999998},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.183},"max":{"1min":1.6779999999999999,"5min":4.3170000000000002,"15min":7.7140000000000004},"last":5.0339999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.93300000000000005},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":2.7919999999999998,"5min":4.3540000000000001,"15min":7.8559999999999999},"last":4.9930000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:11:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.022,"15min":0.97599999999999998},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":3.5,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":3.6320000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0029999999999999,"15min":1.0620000000000001},"min":{"1min":0.32300000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":3.4540000000000002,"5min":5.133,"15min":7.8159999999999998},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4238376960,"data_stored":9228423168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4083445760,"data_stored":8895307795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8321105920,"data_stored":18123014163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:52.483 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:52.484 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:52.484 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:52.485 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:52.485 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:53.755 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.748+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:11:53.756 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.748+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:11:53.756 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.754+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:11:53.757 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.754+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:11:53.757 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.754+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:11:53.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.771+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:11:53.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.771+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:11:53.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.771+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:11:53.776 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.771+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:11:53.776 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:11:53.771+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:12:01.792 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:01.784+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:12:04.778 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.773+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:12:04.779 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.777+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:12:04.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.780+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:12:04.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.780+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:12:04.789 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.786+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:12:04.790 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.786+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:12:04.790 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.786+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:12:04.790 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.786+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:12:04.791 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:04.786+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:12:12.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:13.037 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:12:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:12:13.059 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2313,"stamp":"2023-10-21T05:12:11.825597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29318432509,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2090763,"num_read_kb":8096474,"num_write":2040552,"num_write_kb":8159983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84863,"ondisk_log_size":84863,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20124468,"kb_used_data":16318424,"kb_used_omap":4380,"kb_used_meta":3801635,"kb_avail":262991052,"statfs":{"total":289910292480,"available":269302837248,"internally_reserved":0,"allocated":16710066176,"data_stored":36311620062,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":3892875228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,6,3,7,14,6,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19677184,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4914,"num_read_kb":19604,"num_write":4799,"num_write_kb":19196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002022"},"pg_stats":[{"pgid":"2.7","version":"139'6848","reported_seq":10144,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.789406+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:12:04.789406+0000","last_peered":"2023-10-21T05:12:04.789406+0000","last_clean":"2023-10-21T05:12:04.789406+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:12:04.789406+0000","last_undegraded":"2023-10-21T05:12:04.789406+0000","last_fullsized":"2023-10-21T05:12:04.789406+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'3508","reported_seq":7191,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.743019+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:12:04.743019+0000","last_peered":"2023-10-21T05:12:04.743019+0000","last_clean":"2023-10-21T05:12:04.743019+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:12:04.743019+0000","last_undegraded":"2023-10-21T05:12:04.743019+0000","last_fullsized":"2023-10-21T05:12:04.743019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'10218","reported_seq":15027,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.793314+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:12:04.793314+0000","last_peered":"2023-10-21T05:12:04.793314+0000","last_clean":"2023-10-21T05:12:04.793314+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:12:04.793314+0000","last_undegraded":"2023-10-21T05:12:04.793314+0000","last_fullsized":"2023-10-21T05:12:04.793314+0000","mapping_epoch":11,"log_start":"40'200","ondisk_log_start":"40'200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'6887","reported_seq":11197,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.791431+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:12:04.791431+0000","last_peered":"2023-10-21T05:12:04.791431+0000","last_clean":"2023-10-21T05:12:04.791431+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:12:04.791431+0000","last_undegraded":"2023-10-21T05:12:04.791431+0000","last_fullsized":"2023-10-21T05:12:04.791431+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'512620","reported_seq":1013701,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:11.333709+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:12:11.333709+0000","last_peered":"2023-10-21T05:12:11.333709+0000","last_clean":"2023-10-21T05:12:11.333709+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:12:11.333709+0000","last_undegraded":"2023-10-21T05:12:11.333709+0000","last_fullsized":"2023-10-21T05:12:11.333709+0000","mapping_epoch":14,"log_start":"138'502558","ondisk_log_start":"138'502558","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3172,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 353s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7252062208,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":505635,"num_read_kb":1995324,"num_write":505012,"num_write_kb":2020048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"139'13926","reported_seq":20489,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:10.160012+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:12:10.160012+0000","last_peered":"2023-10-21T05:12:10.160012+0000","last_clean":"2023-10-21T05:12:10.160012+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:12:10.160012+0000","last_undegraded":"2023-10-21T05:12:10.160012+0000","last_fullsized":"2023-10-21T05:12:10.160012+0000","mapping_epoch":11,"log_start":"76'3900","ondisk_log_start":"76'3900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'517665","reported_seq":1025119,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:07.024668+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:12:07.024668+0000","last_peered":"2023-10-21T05:12:07.024668+0000","last_clean":"2023-10-21T05:12:07.024668+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:12:07.024668+0000","last_undegraded":"2023-10-21T05:12:07.024668+0000","last_fullsized":"2023-10-21T05:12:07.024668+0000","mapping_epoch":14,"log_start":"138'507658","ondisk_log_start":"138'507658","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2743,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 332s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7339909120,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":512084,"num_read_kb":2020212,"num_write":510375,"num_write_kb":2041500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3430,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.741100+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:12:04.741100+0000","last_peered":"2023-10-21T05:12:04.741100+0000","last_clean":"2023-10-21T05:12:04.741100+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:12:04.741100+0000","last_undegraded":"2023-10-21T05:12:04.741100+0000","last_fullsized":"2023-10-21T05:12:04.741100+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'541221","reported_seq":1068274,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:11.346677+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:12:11.346677+0000","last_peered":"2023-10-21T05:12:11.346677+0000","last_clean":"2023-10-21T05:12:11.346677+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:12:11.346677+0000","last_undegraded":"2023-10-21T05:12:11.346677+0000","last_fullsized":"2023-10-21T05:12:11.346677+0000","mapping_epoch":14,"log_start":"138'531149","ondisk_log_start":"138'531149","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2708,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 350s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7667904512,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":531916,"num_read_kb":2099352,"num_write":533716,"num_write_kb":2134864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6832,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.740376+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:12:04.740376+0000","last_peered":"2023-10-21T05:12:04.740376+0000","last_clean":"2023-10-21T05:12:04.740376+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:12:04.740376+0000","last_undegraded":"2023-10-21T05:12:04.740376+0000","last_fullsized":"2023-10-21T05:12:04.740376+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'495160","reported_seq":982276,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:07.013834+0000","last_change":"2023-10-21T05:12:03.300021+0000","last_active":"2023-10-21T05:12:07.013834+0000","last_peered":"2023-10-21T05:12:07.013834+0000","last_clean":"2023-10-21T05:12:07.013834+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:12:07.013834+0000","last_undegraded":"2023-10-21T05:12:07.013834+0000","last_fullsized":"2023-10-21T05:12:07.013834+0000","mapping_epoch":14,"log_start":"138'485156","ondisk_log_start":"138'485156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7057960979,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":491669,"num_read_kb":1940516,"num_write":490291,"num_write_kb":1961158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"139'6809","reported_seq":10036,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-21T05:12:04.799332+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:12:04.799332+0000","last_peered":"2023-10-21T05:12:04.799332+0000","last_clean":"2023-10-21T05:12:04.799332+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:12:04.799332+0000","last_undegraded":"2023-10-21T05:12:04.799332+0000","last_fullsized":"2023-10-21T05:12:04.799332+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6809,"log_dups_size":0,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":312,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-21T05:10:20.938843+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:10:20.938843+0000","last_peered":"2023-10-21T05:10:20.938843+0000","last_clean":"2023-10-21T05:10:20.938843+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:10:20.938843+0000","last_undegraded":"2023-10-21T05:10:20.938843+0000","last_fullsized":"2023-10-21T05:10:20.938843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29317836819,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2041304,"num_read_kb":8055404,"num_write":2039394,"num_write_kb":8157570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16706048000,"data_stored":36309864486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40145,"ondisk_log_size":40145,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44619,"ondisk_log_size":44619,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9062800,"kb_used_data":8157892,"kb_used_omap":2268,"kb_used_meta":902627,"kb_avail":85309040,"statfs":{"total":96636764160,"available":87356456960,"internally_reserved":0,"allocated":8353681408,"data_stored":18154639477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":924291052},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.94299999999999995,"15min":0.91000000000000003},"min":{"1min":0.33200000000000002,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.2949999999999999,"5min":3.8170000000000002,"15min":4.8010000000000002},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.84399999999999997,"15min":0.89200000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20599999999999999},"max":{"1min":4.4219999999999997,"5min":4.4219999999999997,"15min":7.2709999999999999},"last":0.94999999999999996}]},{"osd":1,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.087,"15min":1.1200000000000001},"min":{"1min":0.28599999999999998,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":4.3879999999999999,"5min":5.3319999999999999,"15min":27.196999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.23,"15min":1.2210000000000001},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.224},"max":{"1min":9.0020000000000007,"5min":10.253,"15min":27.242999999999999},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359739240,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5337328,"kb_used_data":4004008,"kb_used_omap":1038,"kb_used_meta":1332273,"kb_avail":89034512,"statfs":{"total":96636764160,"available":91171340288,"internally_reserved":0,"allocated":4100104192,"data_stored":8911229001,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1364247816},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:54 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.76800000000000002,"15min":0.85499999999999998},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":1.212,"5min":6.2140000000000004,"15min":10.711},"last":2.8820000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.81799999999999995,"15min":0.95899999999999996},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":1.1839999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:11:54 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.95399999999999996,"15min":1.05},"min":{"1min":0.31,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":4.4969999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":2.516},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.93300000000000005,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":1.492,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739241,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5724340,"kb_used_data":4156524,"kb_used_omap":1073,"kb_used_meta":1566734,"kb_avail":88647500,"statfs":{"total":96636764160,"available":90775040000,"internally_reserved":0,"allocated":4256280576,"data_stored":9245751584,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1604336360},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,7,5,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.183},"max":{"1min":5.0339999999999998,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.96499999999999997,"15min":0.95299999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.9930000000000003,"5min":4.9930000000000003,"15min":7.8559999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:11:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.022,"15min":0.97599999999999998},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":3.5,"5min":5.3179999999999996,"15min":5.7210000000000001},"last":0.44},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0029999999999999,"15min":1.0620000000000001},"min":{"1min":0.32300000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":3.4540000000000002,"5min":5.133,"15min":7.8159999999999998},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4254629888,"data_stored":9244676096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4098727936,"data_stored":8910589971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8352690176,"data_stored":18154598419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:13.061 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:13.061 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:13.062 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:13.062 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:13.062 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:13.085 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:12:13.082+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:12:13.085 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:12:13.082+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:12:13.969 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.966+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:12:13.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.966+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:12:13.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.972+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:12:13.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.972+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:12:13.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.972+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:12:13.980 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.978+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:12:13.983 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.978+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:12:13.984 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.978+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:12:13.984 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.978+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:12:13.984 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:13.978+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:12:25.309 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:12:25.305+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:12:25.310 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:12:25.305+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:12:28.586 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:12:28.582+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:12:33.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:33.610 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:12:33.611 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:12:33.630 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2324,"stamp":"2023-10-21T05:12:31.828995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29349828349,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2098683,"num_read_kb":8128038,"num_write":2048216,"num_write_kb":8190639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85104,"ondisk_log_size":85104,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20520376,"kb_used_data":16378780,"kb_used_omap":4380,"kb_used_meta":4137187,"kb_avail":262595144,"statfs":{"total":289910292480,"available":268897427456,"internally_reserved":0,"allocated":16771870720,"data_stored":36373401953,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":4236480476},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,5,7,9,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19898368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5101,"num_read_kb":20336,"num_write":4856,"num_write_kb":19424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002076"},"pg_stats":[{"pgid":"2.7","version":"140'6868","reported_seq":10173,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.217068+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:12:25.217068+0000","last_peered":"2023-10-21T05:12:25.217068+0000","last_clean":"2023-10-21T05:12:25.217068+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:12:25.217068+0000","last_undegraded":"2023-10-21T05:12:25.217068+0000","last_fullsized":"2023-10-21T05:12:25.217068+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'3518","reported_seq":7206,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.004463+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:12:25.004463+0000","last_peered":"2023-10-21T05:12:25.004463+0000","last_clean":"2023-10-21T05:12:25.004463+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:12:25.004463+0000","last_undegraded":"2023-10-21T05:12:25.004463+0000","last_fullsized":"2023-10-21T05:12:25.004463+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'10247","reported_seq":15069,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.217460+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:12:25.217460+0000","last_peered":"2023-10-21T05:12:25.217460+0000","last_clean":"2023-10-21T05:12:25.217460+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:12:25.217460+0000","last_undegraded":"2023-10-21T05:12:25.217460+0000","last_fullsized":"2023-10-21T05:12:25.217460+0000","mapping_epoch":11,"log_start":"40'200","ondisk_log_start":"40'200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6907","reported_seq":11226,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.216870+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:12:25.216870+0000","last_peered":"2023-10-21T05:12:25.216870+0000","last_clean":"2023-10-21T05:12:25.216870+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:12:25.216870+0000","last_undegraded":"2023-10-21T05:12:25.216870+0000","last_fullsized":"2023-10-21T05:12:25.216870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6907,"log_dups_size":0,"ondisk_log_size":6907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'514636","reported_seq":1017740,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:31.514274+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:12:31.514274+0000","last_peered":"2023-10-21T05:12:31.514274+0000","last_clean":"2023-10-21T05:12:31.514274+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:12:31.514274+0000","last_undegraded":"2023-10-21T05:12:31.514274+0000","last_fullsized":"2023-10-21T05:12:31.514274+0000","mapping_epoch":14,"log_start":"138'504558","ondisk_log_start":"138'504558","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3342,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 373s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7259750400,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":507644,"num_read_kb":2003340,"num_write":506890,"num_write_kb":2027560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'13982","reported_seq":20562,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:31.214358+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:12:31.214358+0000","last_peered":"2023-10-21T05:12:31.214358+0000","last_clean":"2023-10-21T05:12:31.214358+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:12:31.214358+0000","last_undegraded":"2023-10-21T05:12:31.214358+0000","last_fullsized":"2023-10-21T05:12:31.214358+0000","mapping_epoch":11,"log_start":"76'3900","ondisk_log_start":"76'3900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'519736","reported_seq":1029125,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:27.190823+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:12:27.190823+0000","last_peered":"2023-10-21T05:12:27.190823+0000","last_clean":"2023-10-21T05:12:27.190823+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:12:27.190823+0000","last_undegraded":"2023-10-21T05:12:27.190823+0000","last_fullsized":"2023-10-21T05:12:27.190823+0000","mapping_epoch":14,"log_start":"138'509658","ondisk_log_start":"138'509658","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2963,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 352s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7347671040,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":514002,"num_read_kb":2027856,"num_write":512268,"num_write_kb":2049072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3439,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.219943+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:12:25.219943+0000","last_peered":"2023-10-21T05:12:25.219943+0000","last_clean":"2023-10-21T05:12:25.219943+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:12:25.219943+0000","last_undegraded":"2023-10-21T05:12:25.219943+0000","last_fullsized":"2023-10-21T05:12:25.219943+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'543353","reported_seq":1072469,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:31.506563+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:12:31.506563+0000","last_peered":"2023-10-21T05:12:31.506563+0000","last_clean":"2023-10-21T05:12:31.506563+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:12:31.506563+0000","last_undegraded":"2023-10-21T05:12:31.506563+0000","last_fullsized":"2023-10-21T05:12:31.506563+0000","mapping_epoch":14,"log_start":"138'533349","ondisk_log_start":"138'533349","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2839,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 370s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7676207104,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":533966,"num_read_kb":2107516,"num_write":535742,"num_write_kb":2142968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6849,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.217750+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:12:25.217750+0000","last_peered":"2023-10-21T05:12:25.217750+0000","last_clean":"2023-10-21T05:12:25.217750+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:12:25.217750+0000","last_undegraded":"2023-10-21T05:12:25.217750+0000","last_fullsized":"2023-10-21T05:12:25.217750+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'497027","reported_seq":986089,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:27.236560+0000","last_change":"2023-10-21T05:12:26.503616+0000","last_active":"2023-10-21T05:12:27.236560+0000","last_peered":"2023-10-21T05:12:27.236560+0000","last_clean":"2023-10-21T05:12:27.236560+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:12:27.236560+0000","last_undegraded":"2023-10-21T05:12:27.236560+0000","last_fullsized":"2023-10-21T05:12:27.236560+0000","mapping_epoch":14,"log_start":"138'486956","ondisk_log_start":"138'486956","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7065604115,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":493612,"num_read_kb":1948256,"num_write":492158,"num_write_kb":1968626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'6829","reported_seq":10065,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:25.216811+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:12:25.216811+0000","last_peered":"2023-10-21T05:12:25.216811+0000","last_clean":"2023-10-21T05:12:25.216811+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:12:25.216811+0000","last_undegraded":"2023-10-21T05:12:25.216811+0000","last_fullsized":"2023-10-21T05:12:25.216811+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":314,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:14.435195+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:12:14.435195+0000","last_peered":"2023-10-21T05:12:14.435195+0000","last_clean":"2023-10-21T05:12:14.435195+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:12:14.435195+0000","last_undegraded":"2023-10-21T05:12:14.435195+0000","last_fullsized":"2023-10-21T05:12:14.435195+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29349232659,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2049224,"num_read_kb":8086968,"num_write":2047058,"num_write_kb":8188226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16768708608,"data_stored":36372525094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44774,"ondisk_log_size":44774,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9276268,"kb_used_data":8188320,"kb_used_omap":2268,"kb_used_meta":1085667,"kb_avail":85095572,"statfs":{"total":96636764160,"available":87137865728,"internally_reserved":0,"allocated":8384839680,"data_stored":18185790198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":1111724012},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.94299999999999995,"15min":0.91000000000000003},"min":{"1min":0.33200000000000002,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.2949999999999999,"5min":3.8170000000000002,"15min":4.8010000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.84399999999999997,"15min":0.89200000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20599999999999999},"max":{"1min":4.4219999999999997,"5min":4.4219999999999997,"15min":7.2709999999999999},"last":0.38500000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.087,"15min":1.1200000000000001},"min":{"1min":0.28599999999999998,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":4.3879999999999999,"5min":5.3319999999999999,"15min":27.196999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.23,"15min":1.2210000000000001},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.224},"max":{"1min":9.0020000000000007,"5min":10.253,"15min":27.242999999999999},"last":0.745}]}]},{"osd":1,"up_from":8,"seq":34359739244,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5442544,"kb_used_data":4018984,"kb_used_omap":1038,"kb_used_meta":1422513,"kb_avail":88929296,"statfs":{"total":96636764160,"available":91063599104,"internally_reserved":0,"allocated":4115439616,"data_stored":8926556874,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1456653576},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:54 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.76800000000000002,"15min":0.85499999999999998},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":1.212,"5min":6.2140000000000004,"15min":10.711},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.81799999999999995,"15min":0.95899999999999996},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":1.1839999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:11:54 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.95399999999999996,"15min":1.05},"min":{"1min":0.31,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":4.4969999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":1.7549999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.93300000000000005,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":1.492,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":1.74}]}]},{"osd":0,"up_from":8,"seq":34359739245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5801564,"kb_used_data":4171476,"kb_used_omap":1073,"kb_used_meta":1629006,"kb_avail":88570276,"statfs":{"total":96636764160,"available":90695962624,"internally_reserved":0,"allocated":4271591424,"data_stored":9261054881,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1668102888},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.183},"max":{"1min":5.0339999999999998,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":1.248},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.96499999999999997,"15min":0.95299999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.9930000000000003,"5min":4.9930000000000003,"15min":7.8559999999999999},"last":0.252}]},{"osd":2,"last update":"Sat Oct 21 05:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.95599999999999996,"15min":0.98199999999999998},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":4.359,"5min":4.907,"15min":5.7210000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.071,"5min":1.0149999999999999,"15min":1.0569999999999999},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":4.3049999999999997,"5min":5.133,"15min":5.133},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4270628864,"data_stored":9260675072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4114141184,"data_stored":8926003219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8383938560,"data_stored":18185846803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:33.632 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:33.632 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:33.633 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:33.633 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:33.633 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:40.259 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.253+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:12:40.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.253+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:12:40.260 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.253+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:12:40.266 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.263+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:12:40.267 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.263+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:12:40.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.270+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:12:40.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.270+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:12:40.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.270+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:12:40.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.270+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:12:40.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:12:40.270+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:12:53.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:54.192 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:12:54.192 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:12:54.216 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2335,"stamp":"2023-10-21T05:12:53.832916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29386127101,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2107399,"num_read_kb":8162810,"num_write":2057072,"num_write_kb":8226063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85129,"ondisk_log_size":85129,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20907972,"kb_used_data":16445608,"kb_used_omap":4380,"kb_used_meta":4457955,"kb_avail":262207548,"statfs":{"total":289910292480,"available":268500529152,"internally_reserved":0,"allocated":16840302592,"data_stored":36441833825,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":4564946908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,4,3,9,13,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20221952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4785,"num_read_kb":19080,"num_write":4928,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002144"},"pg_stats":[{"pgid":"2.7","version":"140'6900","reported_seq":10215,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.816972+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:12:50.816972+0000","last_peered":"2023-10-21T05:12:50.816972+0000","last_clean":"2023-10-21T05:12:50.816972+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:12:50.816972+0000","last_undegraded":"2023-10-21T05:12:50.816972+0000","last_fullsized":"2023-10-21T05:12:50.816972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'3534","reported_seq":7228,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:46.080404+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:12:46.080404+0000","last_peered":"2023-10-21T05:12:46.080404+0000","last_clean":"2023-10-21T05:12:46.080404+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:12:46.080404+0000","last_undegraded":"2023-10-21T05:12:46.080404+0000","last_fullsized":"2023-10-21T05:12:46.080404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'10295","reported_seq":15132,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.817617+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:12:50.817617+0000","last_peered":"2023-10-21T05:12:50.817617+0000","last_clean":"2023-10-21T05:12:50.817617+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:12:50.817617+0000","last_undegraded":"2023-10-21T05:12:50.817617+0000","last_fullsized":"2023-10-21T05:12:50.817617+0000","mapping_epoch":11,"log_start":"40'200","ondisk_log_start":"40'200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6939","reported_seq":11268,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.816883+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:12:50.816883+0000","last_peered":"2023-10-21T05:12:50.816883+0000","last_clean":"2023-10-21T05:12:50.816883+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:12:50.816883+0000","last_undegraded":"2023-10-21T05:12:50.816883+0000","last_fullsized":"2023-10-21T05:12:50.816883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6939,"log_dups_size":0,"ondisk_log_size":6939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'516711","reported_seq":1021639,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:51.717622+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:12:51.717622+0000","last_peered":"2023-10-21T05:12:51.717622+0000","last_clean":"2023-10-21T05:12:51.717622+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:12:51.717622+0000","last_undegraded":"2023-10-21T05:12:51.717622+0000","last_fullsized":"2023-10-21T05:12:51.717622+0000","mapping_epoch":14,"log_start":"139'506659","ondisk_log_start":"139'506659","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3553,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 393s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7267577856,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":509458,"num_read_kb":2010576,"num_write":508794,"num_write_kb":2035176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'14017","reported_seq":20613,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.816945+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:12:50.816945+0000","last_peered":"2023-10-21T05:12:50.816945+0000","last_clean":"2023-10-21T05:12:50.816945+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:12:50.816945+0000","last_undegraded":"2023-10-21T05:12:50.816945+0000","last_fullsized":"2023-10-21T05:12:50.816945+0000","mapping_epoch":11,"log_start":"76'4000","ondisk_log_start":"76'4000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'522402","reported_seq":1034251,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:52.428703+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:12:52.428703+0000","last_peered":"2023-10-21T05:12:52.428703+0000","last_clean":"2023-10-21T05:12:52.428703+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:12:52.428703+0000","last_undegraded":"2023-10-21T05:12:52.428703+0000","last_fullsized":"2023-10-21T05:12:52.428703+0000","mapping_epoch":14,"log_start":"139'512358","ondisk_log_start":"139'512358","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3157,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 377s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7357939712,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":516452,"num_read_kb":2037644,"num_write":514776,"num_write_kb":2059104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3447,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:45.705844+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:12:45.705844+0000","last_peered":"2023-10-21T05:12:45.705844+0000","last_clean":"2023-10-21T05:12:45.705844+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:12:45.705844+0000","last_undegraded":"2023-10-21T05:12:45.705844+0000","last_fullsized":"2023-10-21T05:12:45.705844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'545499","reported_seq":1076690,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:51.703085+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:12:51.703085+0000","last_peered":"2023-10-21T05:12:51.703085+0000","last_clean":"2023-10-21T05:12:51.703085+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:12:51.703085+0000","last_undegraded":"2023-10-21T05:12:51.703085+0000","last_fullsized":"2023-10-21T05:12:51.703085+0000","mapping_epoch":14,"log_start":"139'535449","ondisk_log_start":"139'535449","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3016,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 390s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7684399104,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":536030,"num_read_kb":2115748,"num_write":537742,"num_write_kb":2150968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6865,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:45.697740+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:12:45.697740+0000","last_peered":"2023-10-21T05:12:45.697740+0000","last_clean":"2023-10-21T05:12:45.697740+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:12:45.697740+0000","last_undegraded":"2023-10-21T05:12:45.697740+0000","last_fullsized":"2023-10-21T05:12:45.697740+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'499471","reported_seq":990934,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:52.434291+0000","last_change":"2023-10-21T05:12:51.668018+0000","last_active":"2023-10-21T05:12:52.434291+0000","last_peered":"2023-10-21T05:12:52.434291+0000","last_clean":"2023-10-21T05:12:52.434291+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:12:52.434291+0000","last_undegraded":"2023-10-21T05:12:52.434291+0000","last_fullsized":"2023-10-21T05:12:52.434291+0000","mapping_epoch":14,"log_start":"138'489456","ondisk_log_start":"138'489456","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7075614739,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":496000,"num_read_kb":1957772,"num_write":494602,"num_write_kb":1978402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'6861","reported_seq":10107,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.816937+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:12:50.816937+0000","last_peered":"2023-10-21T05:12:50.816937+0000","last_clean":"2023-10-21T05:12:50.816937+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:12:50.816937+0000","last_undegraded":"2023-10-21T05:12:50.816937+0000","last_fullsized":"2023-10-21T05:12:50.816937+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":314,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:14.435195+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:12:14.435195+0000","last_peered":"2023-10-21T05:12:14.435195+0000","last_clean":"2023-10-21T05:12:14.435195+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:12:14.435195+0000","last_undegraded":"2023-10-21T05:12:14.435195+0000","last_fullsized":"2023-10-21T05:12:14.435195+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29385531411,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2057940,"num_read_kb":8121740,"num_write":2055914,"num_write_kb":8223650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16837144576,"data_stored":36440961062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40161,"ondisk_log_size":40161,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44869,"ondisk_log_size":44869,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739249,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9464328,"kb_used_data":8219776,"kb_used_omap":2268,"kb_used_meta":1242275,"kb_avail":84907512,"statfs":{"total":96636764160,"available":86945292288,"internally_reserved":0,"allocated":8417050624,"data_stored":18218001142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":1272090604},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.94299999999999995,"15min":0.91000000000000003},"min":{"1min":0.33200000000000002,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.2949999999999999,"5min":3.8170000000000002,"15min":4.8010000000000002},"last":3.3180000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.84399999999999997,"15min":0.89200000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20599999999999999},"max":{"1min":4.4219999999999997,"5min":4.4219999999999997,"15min":7.2709999999999999},"last":3.3370000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.087,"15min":1.1200000000000001},"min":{"1min":0.28599999999999998,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":4.3879999999999999,"5min":5.3319999999999999,"15min":27.196999999999999},"last":1.885},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.23,"15min":1.2210000000000001},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.224},"max":{"1min":9.0020000000000007,"5min":10.253,"15min":27.242999999999999},"last":1.1040000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739249,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5554288,"kb_used_data":4038756,"kb_used_omap":1038,"kb_used_meta":1514481,"kb_avail":88817552,"statfs":{"total":96636764160,"available":90949173248,"internally_reserved":0,"allocated":4135686144,"data_stored":8946803402,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1550828808},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:54 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.76800000000000002,"15min":0.85499999999999998},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":1.212,"5min":6.2140000000000004,"15min":10.711},"last":2.3769999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.81799999999999995,"15min":0.95899999999999996},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":1.1839999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.253}]},{"osd":2,"last update":"Sat Oct 21 05:11:54 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.95399999999999996,"15min":1.05},"min":{"1min":0.31,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":4.4969999999999999,"5min":10.167999999999999,"15min":10.167999999999999},"last":2.7440000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.93300000000000005,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":1.492,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":2.4220000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5889356,"kb_used_data":4187076,"kb_used_omap":1073,"kb_used_meta":1701198,"kb_avail":88482484,"statfs":{"total":96636764160,"available":90606063616,"internally_reserved":0,"allocated":4287565824,"data_stored":9277029281,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1742027496},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,4,11,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.183},"max":{"1min":5.0339999999999998,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.751},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.96499999999999997,"15min":0.95299999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.9930000000000003,"5min":4.9930000000000003,"15min":7.8559999999999999},"last":0.57299999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.95599999999999996,"15min":0.98199999999999998},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":4.359,"5min":4.907,"15min":5.7210000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.0149999999999999,"15min":1.0569999999999999},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":4.3049999999999997,"5min":5.133,"15min":5.133},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4286611456,"data_stored":9276657664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4134412288,"data_stored":8946274323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8416120832,"data_stored":18218029075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:54.218 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:54.218 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:54.219 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:54.219 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:54.220 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:00.162 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:13:00.163 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:13:00.163 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:13:00.163 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:13:00.164 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:13:00.164 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:13:00.164 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:13:00.165 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:13:00.165 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:00.161+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:13:09.545 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:13:09.544+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:13:10.457 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:13:10.455+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:13:10.458 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:13:10.455+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:13:10.458 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:13:10.455+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:13:10.458 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:13:10.455+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:13:14.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:14.764 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:13:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:13:14.791 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2345,"stamp":"2023-10-21T05:13:13.836255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29418145533,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2115152,"num_read_kb":8193694,"num_write":2064894,"num_write_kb":8257351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85289,"ondisk_log_size":85289,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21218728,"kb_used_data":16507920,"kb_used_omap":4380,"kb_used_meta":4706403,"kb_avail":261896792,"statfs":{"total":289910292480,"available":268182315008,"internally_reserved":0,"allocated":16904110080,"data_stored":36505641313,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":4819357660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,3,12,13,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19943424,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4784,"num_read_kb":19056,"num_write":4870,"num_write_kb":19480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001984"},"pg_stats":[{"pgid":"2.7","version":"140'6916","reported_seq":10239,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:11.584257+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:13:11.584257+0000","last_peered":"2023-10-21T05:13:11.584257+0000","last_clean":"2023-10-21T05:13:11.584257+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:13:11.584257+0000","last_undegraded":"2023-10-21T05:13:11.584257+0000","last_fullsized":"2023-10-21T05:13:11.584257+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'3542","reported_seq":7240,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:11.296638+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:13:11.296638+0000","last_peered":"2023-10-21T05:13:11.296638+0000","last_clean":"2023-10-21T05:13:11.296638+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:13:11.296638+0000","last_undegraded":"2023-10-21T05:13:11.296638+0000","last_fullsized":"2023-10-21T05:13:11.296638+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'10319","reported_seq":15168,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:11.591834+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:13:11.591834+0000","last_peered":"2023-10-21T05:13:11.591834+0000","last_clean":"2023-10-21T05:13:11.591834+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:13:11.591834+0000","last_undegraded":"2023-10-21T05:13:11.591834+0000","last_fullsized":"2023-10-21T05:13:11.591834+0000","mapping_epoch":11,"log_start":"42'300","ondisk_log_start":"42'300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6955","reported_seq":11292,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:11.584308+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:13:11.584308+0000","last_peered":"2023-10-21T05:13:11.584308+0000","last_clean":"2023-10-21T05:13:11.584308+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:13:11.584308+0000","last_undegraded":"2023-10-21T05:13:11.584308+0000","last_fullsized":"2023-10-21T05:13:11.584308+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'518835","reported_seq":1025705,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:11.817071+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:13:11.817071+0000","last_peered":"2023-10-21T05:13:11.817071+0000","last_clean":"2023-10-21T05:13:11.817071+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:13:11.817071+0000","last_undegraded":"2023-10-21T05:13:11.817071+0000","last_fullsized":"2023-10-21T05:13:11.817071+0000","mapping_epoch":14,"log_start":"139'508760","ondisk_log_start":"139'508760","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3735,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 413s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7275659264,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":511383,"num_read_kb":2018256,"num_write":510770,"num_write_kb":2043080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'14070","reported_seq":20682,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:11.297066+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:13:11.297066+0000","last_peered":"2023-10-21T05:13:11.297066+0000","last_clean":"2023-10-21T05:13:11.297066+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:13:11.297066+0000","last_undegraded":"2023-10-21T05:13:11.297066+0000","last_fullsized":"2023-10-21T05:13:11.297066+0000","mapping_epoch":11,"log_start":"76'4000","ondisk_log_start":"76'4000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'524520","reported_seq":1038275,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:12.533604+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:13:12.533604+0000","last_peered":"2023-10-21T05:13:12.533604+0000","last_clean":"2023-10-21T05:13:12.533604+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:13:12.533604+0000","last_undegraded":"2023-10-21T05:13:12.533604+0000","last_fullsized":"2023-10-21T05:13:12.533604+0000","mapping_epoch":14,"log_start":"139'514458","ondisk_log_start":"139'514458","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3319,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 397s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7366078464,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":518343,"num_read_kb":2045168,"num_write":516764,"num_write_kb":2067056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3456,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:06.212582+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:13:06.212582+0000","last_peered":"2023-10-21T05:13:06.212582+0000","last_clean":"2023-10-21T05:13:06.212582+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:13:06.212582+0000","last_undegraded":"2023-10-21T05:13:06.212582+0000","last_fullsized":"2023-10-21T05:13:06.212582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'547622","reported_seq":1080885,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:11.793257+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:13:11.793257+0000","last_peered":"2023-10-21T05:13:11.793257+0000","last_clean":"2023-10-21T05:13:11.793257+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:13:11.793257+0000","last_undegraded":"2023-10-21T05:13:11.793257+0000","last_fullsized":"2023-10-21T05:13:11.793257+0000","mapping_epoch":14,"log_start":"139'537550","ondisk_log_start":"139'537550","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3173,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 410s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7692558336,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":538091,"num_read_kb":2123968,"num_write":539736,"num_write_kb":2158944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6881,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:06.179547+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:13:06.179547+0000","last_peered":"2023-10-21T05:13:06.179547+0000","last_clean":"2023-10-21T05:13:06.179547+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:13:06.179547+0000","last_undegraded":"2023-10-21T05:13:06.179547+0000","last_fullsized":"2023-10-21T05:13:06.179547+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'501335","reported_seq":994683,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:12.541935+0000","last_change":"2023-10-21T05:13:09.022706+0000","last_active":"2023-10-21T05:13:12.541935+0000","last_peered":"2023-10-21T05:13:12.541935+0000","last_clean":"2023-10-21T05:13:12.541935+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:13:12.541935+0000","last_undegraded":"2023-10-21T05:13:12.541935+0000","last_fullsized":"2023-10-21T05:13:12.541935+0000","mapping_epoch":14,"log_start":"139'491256","ondisk_log_start":"139'491256","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7083253779,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":497876,"num_read_kb":1965232,"num_write":496466,"num_write_kb":1985858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'6877","reported_seq":10132,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:13:11.586634+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:13:11.586634+0000","last_peered":"2023-10-21T05:13:11.586634+0000","last_clean":"2023-10-21T05:13:11.586634+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:13:11.586634+0000","last_undegraded":"2023-10-21T05:13:11.586634+0000","last_fullsized":"2023-10-21T05:13:11.586634+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":314,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:14.435195+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:12:14.435195+0000","last_peered":"2023-10-21T05:12:14.435195+0000","last_clean":"2023-10-21T05:12:14.435195+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:12:14.435195+0000","last_undegraded":"2023-10-21T05:12:14.435195+0000","last_fullsized":"2023-10-21T05:12:14.435195+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29417549843,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2065693,"num_read_kb":8152624,"num_write":2063736,"num_write_kb":8254938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16901230592,"data_stored":36505047078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40288,"ondisk_log_size":40288,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44902,"ondisk_log_size":44902,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739253,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9617456,"kb_used_data":8250728,"kb_used_omap":2268,"kb_used_meta":1364451,"kb_avail":84754384,"statfs":{"total":96636764160,"available":86788489216,"internally_reserved":0,"allocated":8448745472,"data_stored":18249695990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":1397198828},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":0.96499999999999997,"15min":0.92700000000000005},"min":{"1min":0.26300000000000001,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.6960000000000002,"5min":3.8170000000000002,"15min":4.8010000000000002},"last":3.4900000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84699999999999998,"15min":0.88100000000000001},"min":{"1min":0.23300000000000001,"5min":0.21099999999999999,"15min":0.20599999999999999},"max":{"1min":3.3370000000000002,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.435}]},{"osd":1,"last update":"Sat Oct 21 05:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.069,"15min":1.0920000000000001},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.19900000000000001},"max":{"1min":3.6600000000000001,"5min":5.3319999999999999,"15min":27.196999999999999},"last":3.4249999999999998},{"interface":"front","average":{"1min":1.159,"5min":1.228,"15min":1.2310000000000001},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":3.7650000000000001,"5min":10.253,"15min":27.242999999999999},"last":3.4580000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739253,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5637788,"kb_used_data":4054292,"kb_used_omap":1038,"kb_used_meta":1582449,"kb_avail":88734052,"statfs":{"total":96636764160,"available":90863669248,"internally_reserved":0,"allocated":4151595008,"data_stored":8962712266,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1620428040},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,1,4,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.83599999999999997,"15min":0.85199999999999998},"min":{"1min":0.23400000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":2.8820000000000001,"5min":6.2140000000000004,"15min":10.711},"last":3.04},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.80600000000000005,"15min":0.94099999999999995},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.20599999999999999},"max":{"1min":2.6619999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.32000000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.93400000000000005,"15min":1.0780000000000001},"min":{"1min":0.34200000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":2.8399999999999999,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.056,"5min":0.93300000000000005,"15min":1.127},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.5899999999999999,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5963484,"kb_used_data":4202900,"kb_used_omap":1073,"kb_used_meta":1759502,"kb_avail":88408356,"statfs":{"total":96636764160,"available":90530156544,"internally_reserved":0,"allocated":4303769600,"data_stored":9293233057,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1801730792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.183},"max":{"1min":3.944,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96399999999999997,"15min":0.95899999999999996},"min":{"1min":0.252,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9809999999999999,"5min":4.9930000000000003,"15min":7.8559999999999999},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.95599999999999996,"15min":0.98199999999999998},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":4.359,"5min":4.907,"15min":5.7210000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.071,"5min":1.0149999999999999,"15min":1.0569999999999999},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":4.3049999999999997,"5min":5.133,"15min":5.133},"last":1.891}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4302872576,"data_stored":9292918784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4150190080,"data_stored":8962052115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8448167936,"data_stored":18250076179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:14.793 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:14.793 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:14.794 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:14.794 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:14.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:13:15.339 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:13:15.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:13:15.896 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:13:15.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:13:16.464 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:13:16.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:13:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:13:17.021 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:35.332 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:35.324+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:13:35.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:35.324+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:13:35.336 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:35.325+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:13:35.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:35.325+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:13:35.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:35.325+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:13:37.023 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:37.564 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:13:37.565 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:13:37.590 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2357,"stamp":"2023-10-21T05:13:35.840080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29448722173,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2122607,"num_read_kb":8223450,"num_write":2072354,"num_write_kb":8287191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85191,"ondisk_log_size":85191,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21635572,"kb_used_data":16575580,"kb_used_omap":4380,"kb_used_meta":5055587,"kb_avail":261479948,"statfs":{"total":289910292480,"available":267755466752,"internally_reserved":0,"allocated":16973393920,"data_stored":36574902371,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":5176922076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,9,6,14,12,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15749120,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3824,"num_read_kb":15260,"num_write":3841,"num_write_kb":15364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.408535"},"pg_stats":[{"pgid":"2.7","version":"141'6934","reported_seq":10266,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:32.190279+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:13:32.190279+0000","last_peered":"2023-10-21T05:13:32.190279+0000","last_clean":"2023-10-21T05:13:32.190279+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:13:32.190279+0000","last_undegraded":"2023-10-21T05:13:32.190279+0000","last_fullsized":"2023-10-21T05:13:32.190279+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6934,"log_dups_size":0,"ondisk_log_size":6934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'3551","reported_seq":7254,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:31.776760+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:13:31.776760+0000","last_peered":"2023-10-21T05:13:31.776760+0000","last_clean":"2023-10-21T05:13:31.776760+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:13:31.776760+0000","last_undegraded":"2023-10-21T05:13:31.776760+0000","last_fullsized":"2023-10-21T05:13:31.776760+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3551,"log_dups_size":0,"ondisk_log_size":3551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'10346","reported_seq":15209,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:32.207798+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:13:32.207798+0000","last_peered":"2023-10-21T05:13:32.207798+0000","last_clean":"2023-10-21T05:13:32.207798+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:13:32.207798+0000","last_undegraded":"2023-10-21T05:13:32.207798+0000","last_fullsized":"2023-10-21T05:13:32.207798+0000","mapping_epoch":11,"log_start":"42'300","ondisk_log_start":"42'300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'6973","reported_seq":11320,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:32.196842+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:13:32.196842+0000","last_peered":"2023-10-21T05:13:32.196842+0000","last_clean":"2023-10-21T05:13:32.196842+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:13:32.196842+0000","last_undegraded":"2023-10-21T05:13:32.196842+0000","last_fullsized":"2023-10-21T05:13:32.196842+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":6973,"log_dups_size":0,"ondisk_log_size":6973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'520817","reported_seq":1029548,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:32.066895+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:13:32.066895+0000","last_peered":"2023-10-21T05:13:32.066895+0000","last_clean":"2023-10-21T05:13:32.066895+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:13:32.066895+0000","last_undegraded":"2023-10-21T05:13:32.066895+0000","last_fullsized":"2023-10-21T05:13:32.066895+0000","mapping_epoch":14,"log_start":"139'510760","ondisk_log_start":"139'510760","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3959,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 434s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7283056640,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":513224,"num_read_kb":2025604,"num_write":512569,"num_write_kb":2050276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'14122","reported_seq":20752,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:31.777495+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:13:31.777495+0000","last_peered":"2023-10-21T05:13:31.777495+0000","last_clean":"2023-10-21T05:13:31.777495+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:13:31.777495+0000","last_undegraded":"2023-10-21T05:13:31.777495+0000","last_fullsized":"2023-10-21T05:13:31.777495+0000","mapping_epoch":11,"log_start":"76'4100","ondisk_log_start":"76'4100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'526493","reported_seq":1042068,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:32.725858+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:13:32.725858+0000","last_peered":"2023-10-21T05:13:32.725858+0000","last_clean":"2023-10-21T05:13:32.725858+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:13:32.725858+0000","last_undegraded":"2023-10-21T05:13:32.725858+0000","last_fullsized":"2023-10-21T05:13:32.725858+0000","mapping_epoch":14,"log_start":"139'516458","ondisk_log_start":"139'516458","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3521,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 417s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7373455360,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":520152,"num_read_kb":2052388,"num_write":518568,"num_write_kb":2074272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3467,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:31.780421+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:13:31.780421+0000","last_peered":"2023-10-21T05:13:31.780421+0000","last_clean":"2023-10-21T05:13:31.780421+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:13:31.780421+0000","last_undegraded":"2023-10-21T05:13:31.780421+0000","last_fullsized":"2023-10-21T05:13:31.780421+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'549793","reported_seq":1085089,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:32.065803+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:13:32.065803+0000","last_peered":"2023-10-21T05:13:32.065803+0000","last_clean":"2023-10-21T05:13:32.065803+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:13:32.065803+0000","last_undegraded":"2023-10-21T05:13:32.065803+0000","last_fullsized":"2023-10-21T05:13:32.065803+0000","mapping_epoch":14,"log_start":"139'539750","ondisk_log_start":"139'539750","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 431s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7700848640,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":540109,"num_read_kb":2132016,"num_write":541759,"num_write_kb":2167036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6902,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:31.791417+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:13:31.791417+0000","last_peered":"2023-10-21T05:13:31.791417+0000","last_clean":"2023-10-21T05:13:31.791417+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:13:31.791417+0000","last_undegraded":"2023-10-21T05:13:31.791417+0000","last_fullsized":"2023-10-21T05:13:31.791417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'503169","reported_seq":998308,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:32.748999+0000","last_change":"2023-10-21T05:13:29.946456+0000","last_active":"2023-10-21T05:13:32.748999+0000","last_peered":"2023-10-21T05:13:32.748999+0000","last_clean":"2023-10-21T05:13:32.748999+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:13:32.748999+0000","last_undegraded":"2023-10-21T05:13:32.748999+0000","last_fullsized":"2023-10-21T05:13:32.748999+0000","mapping_epoch":14,"log_start":"139'493156","ondisk_log_start":"139'493156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7090765843,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":499663,"num_read_kb":1972372,"num_write":498300,"num_write_kb":1993194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'6895","reported_seq":10159,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:32.201948+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:13:32.201948+0000","last_peered":"2023-10-21T05:13:32.201948+0000","last_clean":"2023-10-21T05:13:32.201948+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:13:32.201948+0000","last_undegraded":"2023-10-21T05:13:32.201948+0000","last_fullsized":"2023-10-21T05:13:32.201948+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":315,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:22.144525+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:12:22.144525+0000","last_peered":"2023-10-21T05:12:22.144525+0000","last_clean":"2023-10-21T05:12:22.144525+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:12:22.144525+0000","last_undegraded":"2023-10-21T05:12:22.144525+0000","last_fullsized":"2023-10-21T05:12:22.144525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29448126483,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2073148,"num_read_kb":8182380,"num_write":2071196,"num_write_kb":8284778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16970321920,"data_stored":36574138406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":44944,"ondisk_log_size":44944,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739258,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9851944,"kb_used_data":8287968,"kb_used_omap":2268,"kb_used_meta":1561699,"kb_avail":84519896,"statfs":{"total":96636764160,"available":86548373504,"internally_reserved":0,"allocated":8486879232,"data_stored":18287822156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":1599180780},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":0.96499999999999997,"15min":0.92700000000000005},"min":{"1min":0.26300000000000001,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.6960000000000002,"5min":3.8170000000000002,"15min":4.8010000000000002},"last":2.3639999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84699999999999998,"15min":0.88100000000000001},"min":{"1min":0.23300000000000001,"5min":0.21099999999999999,"15min":0.20599999999999999},"max":{"1min":3.3370000000000002,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.503}]},{"osd":1,"last update":"Sat Oct 21 05:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.069,"15min":1.0920000000000001},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.19900000000000001},"max":{"1min":3.6600000000000001,"5min":5.3319999999999999,"15min":27.196999999999999},"last":2.2919999999999998},{"interface":"front","average":{"1min":1.159,"5min":1.228,"15min":1.2310000000000001},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":3.7650000000000001,"5min":10.253,"15min":27.242999999999999},"last":2.3279999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739257,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5750080,"kb_used_data":4068792,"kb_used_omap":1038,"kb_used_meta":1680241,"kb_avail":88621760,"statfs":{"total":96636764160,"available":90748682240,"internally_reserved":0,"allocated":4166443008,"data_stored":8977552672,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1720567048},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.83599999999999997,"15min":0.85199999999999998},"min":{"1min":0.23400000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":2.8820000000000001,"5min":6.2140000000000004,"15min":10.711},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.80600000000000005,"15min":0.94099999999999995},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.20599999999999999},"max":{"1min":2.6619999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.22900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.93400000000000005,"15min":1.0780000000000001},"min":{"1min":0.34200000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":2.8399999999999999,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.056,"5min":0.93300000000000005,"15min":1.127},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.5899999999999999,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.34799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6033548,"kb_used_data":4218820,"kb_used_omap":1073,"kb_used_meta":1813646,"kb_avail":88338292,"statfs":{"total":96636764160,"available":90458411008,"internally_reserved":0,"allocated":4320071680,"data_stored":9309527543,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1857174248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,6,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.183},"max":{"1min":3.944,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":3.5369999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96399999999999997,"15min":0.95899999999999996},"min":{"1min":0.252,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9809999999999999,"5min":4.9930000000000003,"15min":7.8559999999999999},"last":3.4889999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:13:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.95199999999999996,"15min":0.97099999999999997},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":3.5070000000000001,"5min":4.907,"15min":5.7210000000000001},"last":0.222},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.92300000000000004,"15min":1.0389999999999999},"min":{"1min":0.23400000000000001,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.1699999999999999,"5min":5.133,"15min":5.133},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4318539776,"data_stored":9308585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4165095424,"data_stored":8976957459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8486686720,"data_stored":18288594963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:37.592 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:37.592 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:37.592 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:37.593 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:37.593 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:37.785 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:13:37.785 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:13:37.786 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:13:37.786 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:13:37.787 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:13:37.787 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:13:37.787 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:13:37.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:13:37.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:13:37.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:37.781+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:13:46.950 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.947+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:13:46.951 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.947+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:13:46.955 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.953+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:13:46.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:13:46.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:13:46.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:13:46.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:13:46.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:13:46.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:13:46.968 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:46.964+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:13:48.301 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.297+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:13:48.301 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.297+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:13:48.314 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.310+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:13:48.314 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:13:48.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:13:48.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:13:48.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:13:48.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:13:48.316 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:13:48.316 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:13:48.311+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:13:57.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:13:58.125 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:13:58.151 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2368,"stamp":"2023-10-21T05:13:57.843986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29484635901,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2131363,"num_read_kb":8258326,"num_write":2081118,"num_write_kb":8322247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85498,"ondisk_log_size":85498,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21997188,"kb_used_data":16641772,"kb_used_omap":4380,"kb_used_meta":5351011,"kb_avail":261118332,"statfs":{"total":289910292480,"available":267385171968,"internally_reserved":0,"allocated":17041174528,"data_stored":36642682979,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":5479436252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,7,6,17,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19894272,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4731,"num_read_kb":18852,"num_write":4856,"num_write_kb":19424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002074"},"pg_stats":[{"pgid":"2.7","version":"141'6965","reported_seq":10305,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.257220+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:13:52.257220+0000","last_peered":"2023-10-21T05:13:52.257220+0000","last_clean":"2023-10-21T05:13:52.257220+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:13:52.257220+0000","last_undegraded":"2023-10-21T05:13:52.257220+0000","last_fullsized":"2023-10-21T05:13:52.257220+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'3567","reported_seq":7274,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:51.456302+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:13:51.456302+0000","last_peered":"2023-10-21T05:13:51.456302+0000","last_clean":"2023-10-21T05:13:51.456302+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:13:51.456302+0000","last_undegraded":"2023-10-21T05:13:51.456302+0000","last_fullsized":"2023-10-21T05:13:51.456302+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'10392","reported_seq":15267,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.257668+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:13:52.257668+0000","last_peered":"2023-10-21T05:13:52.257668+0000","last_clean":"2023-10-21T05:13:52.257668+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:13:52.257668+0000","last_undegraded":"2023-10-21T05:13:52.257668+0000","last_fullsized":"2023-10-21T05:13:52.257668+0000","mapping_epoch":11,"log_start":"42'300","ondisk_log_start":"42'300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'7004","reported_seq":11359,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.257251+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:13:52.257251+0000","last_peered":"2023-10-21T05:13:52.257251+0000","last_clean":"2023-10-21T05:13:52.257251+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:13:52.257251+0000","last_undegraded":"2023-10-21T05:13:52.257251+0000","last_fullsized":"2023-10-21T05:13:52.257251+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7004,"log_dups_size":0,"ondisk_log_size":7004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'523410","reported_seq":1034548,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:57.326579+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:13:57.326579+0000","last_peered":"2023-10-21T05:13:57.326579+0000","last_clean":"2023-10-21T05:13:57.326579+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:13:57.326579+0000","last_undegraded":"2023-10-21T05:13:57.326579+0000","last_fullsized":"2023-10-21T05:13:57.326579+0000","mapping_epoch":14,"log_start":"140'513360","ondisk_log_start":"140'513360","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4181,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 459s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7292928000,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":515613,"num_read_kb":2035116,"num_write":514978,"num_write_kb":2059912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'14154","reported_seq":20800,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:57.029177+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:13:57.029177+0000","last_peered":"2023-10-21T05:13:57.029177+0000","last_clean":"2023-10-21T05:13:57.029177+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:13:57.029177+0000","last_undegraded":"2023-10-21T05:13:57.029177+0000","last_fullsized":"2023-10-21T05:13:57.029177+0000","mapping_epoch":11,"log_start":"76'4100","ondisk_log_start":"76'4100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'528525","reported_seq":1046000,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:52.898590+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:13:52.898590+0000","last_peered":"2023-10-21T05:13:52.898590+0000","last_clean":"2023-10-21T05:13:52.898590+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:13:52.898590+0000","last_undegraded":"2023-10-21T05:13:52.898590+0000","last_fullsized":"2023-10-21T05:13:52.898590+0000","mapping_epoch":14,"log_start":"140'518459","ondisk_log_start":"140'518459","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3687,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 437s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7381217280,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":522034,"num_read_kb":2059896,"num_write":520465,"num_write_kb":2081860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3475,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.259438+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:13:52.259438+0000","last_peered":"2023-10-21T05:13:52.259438+0000","last_clean":"2023-10-21T05:13:52.259438+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:13:52.259438+0000","last_undegraded":"2023-10-21T05:13:52.259438+0000","last_fullsized":"2023-10-21T05:13:52.259438+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'552527","reported_seq":1090426,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:57.330369+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:13:57.330369+0000","last_peered":"2023-10-21T05:13:57.330369+0000","last_clean":"2023-10-21T05:13:57.330369+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:13:57.330369+0000","last_undegraded":"2023-10-21T05:13:57.330369+0000","last_fullsized":"2023-10-21T05:13:57.330369+0000","mapping_epoch":14,"log_start":"140'542450","ondisk_log_start":"140'542450","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3519,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 456s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7711498240,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":542701,"num_read_kb":2142328,"num_write":544355,"num_write_kb":2177420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6918,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.260839+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:13:52.260839+0000","last_peered":"2023-10-21T05:13:52.260839+0000","last_clean":"2023-10-21T05:13:52.260839+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:13:52.260839+0000","last_undegraded":"2023-10-21T05:13:52.260839+0000","last_fullsized":"2023-10-21T05:13:52.260839+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'505031","reported_seq":1002068,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.865963+0000","last_change":"2023-10-21T05:13:49.124517+0000","last_active":"2023-10-21T05:13:52.865963+0000","last_peered":"2023-10-21T05:13:52.865963+0000","last_clean":"2023-10-21T05:13:52.865963+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:13:52.865963+0000","last_undegraded":"2023-10-21T05:13:52.865963+0000","last_fullsized":"2023-10-21T05:13:52.865963+0000","mapping_epoch":14,"log_start":"139'494956","ondisk_log_start":"139'494956","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7098396691,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":501556,"num_read_kb":1979916,"num_write":500162,"num_write_kb":2000642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'6926","reported_seq":10198,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:52.257546+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:13:52.257546+0000","last_peered":"2023-10-21T05:13:52.257546+0000","last_clean":"2023-10-21T05:13:52.257546+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:13:52.257546+0000","last_undegraded":"2023-10-21T05:13:52.257546+0000","last_fullsized":"2023-10-21T05:13:52.257546+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6926,"log_dups_size":0,"ondisk_log_size":6926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":315,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:22.144525+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:12:22.144525+0000","last_peered":"2023-10-21T05:12:22.144525+0000","last_clean":"2023-10-21T05:12:22.144525+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:12:22.144525+0000","last_undegraded":"2023-10-21T05:12:22.144525+0000","last_fullsized":"2023-10-21T05:12:22.144525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29484040211,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2081904,"num_read_kb":8217256,"num_write":2079960,"num_write_kb":8319834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17037852672,"data_stored":36641669158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40268,"ondisk_log_size":40268,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45131,"ondisk_log_size":45131,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739262,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10030100,"kb_used_data":8319180,"kb_used_omap":2268,"kb_used_meta":1708643,"kb_avail":84341740,"statfs":{"total":96636764160,"available":86365941760,"internally_reserved":0,"allocated":8518840320,"data_stored":18319783244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":1749651436},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":0.96499999999999997,"15min":0.92700000000000005},"min":{"1min":0.26300000000000001,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.6960000000000002,"5min":3.8170000000000002,"15min":4.8010000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84699999999999998,"15min":0.88100000000000001},"min":{"1min":0.23300000000000001,"5min":0.21099999999999999,"15min":0.20599999999999999},"max":{"1min":3.3370000000000002,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":1.706}]},{"osd":1,"last update":"Sat Oct 21 05:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.069,"15min":1.0920000000000001},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.19900000000000001},"max":{"1min":3.6600000000000001,"5min":5.3319999999999999,"15min":27.196999999999999},"last":2.0009999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.228,"15min":1.2310000000000001},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":3.7650000000000001,"5min":10.253,"15min":27.242999999999999},"last":0.82799999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739261,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5819116,"kb_used_data":4083748,"kb_used_omap":1038,"kb_used_meta":1734321,"kb_avail":88552724,"statfs":{"total":96636764160,"available":90677989376,"internally_reserved":0,"allocated":4181757952,"data_stored":8992867616,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1775944968},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,0,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.83599999999999997,"15min":0.85199999999999998},"min":{"1min":0.23400000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":2.8820000000000001,"5min":6.2140000000000004,"15min":10.711},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.80600000000000005,"15min":0.94099999999999995},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.20599999999999999},"max":{"1min":2.6619999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.84199999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.93400000000000005,"15min":1.0780000000000001},"min":{"1min":0.34200000000000003,"5min":0.21099999999999999,"15min":0.185},"max":{"1min":2.8399999999999999,"5min":4.4969999999999999,"15min":10.167999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.056,"5min":0.93300000000000005,"15min":1.127},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.5899999999999999,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":1.056}]}]},{"osd":0,"up_from":8,"seq":34359739262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6147972,"kb_used_data":4238844,"kb_used_omap":1073,"kb_used_meta":1908046,"kb_avail":88223868,"statfs":{"total":96636764160,"available":90341240832,"internally_reserved":0,"allocated":4340576256,"data_stored":9330032119,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":1953839848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,6,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.183},"max":{"1min":3.944,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96399999999999997,"15min":0.95899999999999996},"min":{"1min":0.252,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9809999999999999,"5min":4.9930000000000003,"15min":7.8559999999999999},"last":0.374}]},{"osd":2,"last update":"Sat Oct 21 05:13:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.95199999999999996,"15min":0.97099999999999997},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":3.5070000000000001,"5min":4.907,"15min":5.7210000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.92300000000000004,"15min":1.0389999999999999},"min":{"1min":0.23400000000000001,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.1699999999999999,"5min":5.133,"15min":5.133},"last":0.44600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4339032064,"data_stored":9329078272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4180488192,"data_stored":8992350227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8518332416,"data_stored":18320240659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:58.153 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:58.154 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:58.154 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:58.154 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:58.155 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:08.380 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:14:08.377+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:14:08.380 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:14:08.377+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:14:10.065 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:10.061+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:14:10.072 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:10.068+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:14:17.302 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:17.300+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:14:17.302 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:17.300+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:14:17.303 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:17.300+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:14:18.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:18.672 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:14:18.673 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:14:18.699 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2378,"stamp":"2023-10-21T05:14:17.847317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29515810557,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2139097,"num_read_kb":8289134,"num_write":2088718,"num_write_kb":8352647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85355,"ondisk_log_size":85355,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22322780,"kb_used_data":16702660,"kb_used_omap":4380,"kb_used_meta":5615715,"kb_avail":260792740,"statfs":{"total":289910292480,"available":267051765760,"internally_reserved":0,"allocated":17103523840,"data_stored":36705032291,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4485156,"internal_metadata":5750493148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,0,4,10,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19804160,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4959,"num_read_kb":19748,"num_write":4830,"num_write_kb":19320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001995"},"pg_stats":[{"pgid":"2.7","version":"141'6994","reported_seq":10343,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:12.736927+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:14:12.736927+0000","last_peered":"2023-10-21T05:14:12.736927+0000","last_clean":"2023-10-21T05:14:12.736927+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:14:12.736927+0000","last_undegraded":"2023-10-21T05:14:12.736927+0000","last_fullsized":"2023-10-21T05:14:12.736927+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'3582","reported_seq":7293,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:11.157159+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:14:11.157159+0000","last_peered":"2023-10-21T05:14:11.157159+0000","last_clean":"2023-10-21T05:14:11.157159+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:14:11.157159+0000","last_undegraded":"2023-10-21T05:14:11.157159+0000","last_fullsized":"2023-10-21T05:14:11.157159+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'10435","reported_seq":15322,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:12.736993+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:14:12.736993+0000","last_peered":"2023-10-21T05:14:12.736993+0000","last_clean":"2023-10-21T05:14:12.736993+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:14:12.736993+0000","last_undegraded":"2023-10-21T05:14:12.736993+0000","last_fullsized":"2023-10-21T05:14:12.736993+0000","mapping_epoch":11,"log_start":"42'400","ondisk_log_start":"42'400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'7032","reported_seq":11395,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:12.736981+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:14:12.736981+0000","last_peered":"2023-10-21T05:14:12.736981+0000","last_clean":"2023-10-21T05:14:12.736981+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:14:12.736981+0000","last_undegraded":"2023-10-21T05:14:12.736981+0000","last_fullsized":"2023-10-21T05:14:12.736981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'525476","reported_seq":1038581,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:17.479742+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:14:17.479742+0000","last_peered":"2023-10-21T05:14:17.479742+0000","last_clean":"2023-10-21T05:14:17.479742+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:14:17.479742+0000","last_undegraded":"2023-10-21T05:14:17.479742+0000","last_fullsized":"2023-10-21T05:14:17.479742+0000","mapping_epoch":14,"log_start":"140'515460","ondisk_log_start":"140'515460","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4370,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 479s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7300759552,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":517563,"num_read_kb":2042860,"num_write":516888,"num_write_kb":2067552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'14230","reported_seq":20892,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:16.799757+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:14:16.799757+0000","last_peered":"2023-10-21T05:14:16.799757+0000","last_clean":"2023-10-21T05:14:16.799757+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:14:16.799757+0000","last_undegraded":"2023-10-21T05:14:16.799757+0000","last_fullsized":"2023-10-21T05:14:16.799757+0000","mapping_epoch":11,"log_start":"78'4200","ondisk_log_start":"78'4200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'530521","reported_seq":1049991,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:13.022803+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:14:13.022803+0000","last_peered":"2023-10-21T05:14:13.022803+0000","last_clean":"2023-10-21T05:14:13.022803+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:14:13.022803+0000","last_undegraded":"2023-10-21T05:14:13.022803+0000","last_fullsized":"2023-10-21T05:14:13.022803+0000","mapping_epoch":14,"log_start":"140'520459","ondisk_log_start":"140'520459","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3895,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 458s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7388688384,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":524012,"num_read_kb":2067796,"num_write":522289,"num_write_kb":2089156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3483,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:12.740490+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:14:12.740490+0000","last_peered":"2023-10-21T05:14:12.740490+0000","last_clean":"2023-10-21T05:14:12.740490+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:14:12.740490+0000","last_undegraded":"2023-10-21T05:14:12.740490+0000","last_fullsized":"2023-10-21T05:14:12.740490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'554651","reported_seq":1094505,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:17.466444+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:14:17.466444+0000","last_peered":"2023-10-21T05:14:17.466444+0000","last_clean":"2023-10-21T05:14:17.466444+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:14:17.466444+0000","last_undegraded":"2023-10-21T05:14:17.466444+0000","last_fullsized":"2023-10-21T05:14:17.466444+0000","mapping_epoch":14,"log_start":"140'544650","ondisk_log_start":"140'544650","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3653,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 476s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7719788544,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":544640,"num_read_kb":2150044,"num_write":546369,"num_write_kb":2185476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6935,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:12.743431+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:14:12.743431+0000","last_peered":"2023-10-21T05:14:12.743431+0000","last_clean":"2023-10-21T05:14:12.743431+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:14:12.743431+0000","last_undegraded":"2023-10-21T05:14:12.743431+0000","last_fullsized":"2023-10-21T05:14:12.743431+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'506883","reported_seq":1005796,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:13.035788+0000","last_change":"2023-10-21T05:14:11.433841+0000","last_active":"2023-10-21T05:14:13.035788+0000","last_peered":"2023-10-21T05:14:13.035788+0000","last_clean":"2023-10-21T05:14:13.035788+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:14:13.035788+0000","last_undegraded":"2023-10-21T05:14:13.035788+0000","last_fullsized":"2023-10-21T05:14:13.035788+0000","mapping_epoch":14,"log_start":"140'496856","ondisk_log_start":"140'496856","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7105978387,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":503423,"num_read_kb":1987364,"num_write":502014,"num_write_kb":2008050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'6954","reported_seq":10234,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:12.736923+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:14:12.736923+0000","last_peered":"2023-10-21T05:14:12.736923+0000","last_clean":"2023-10-21T05:14:12.736923+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:14:12.736923+0000","last_undegraded":"2023-10-21T05:14:12.736923+0000","last_fullsized":"2023-10-21T05:14:12.736923+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6954,"log_dups_size":0,"ondisk_log_size":6954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":315,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-21T05:12:22.144525+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:12:22.144525+0000","last_peered":"2023-10-21T05:12:22.144525+0000","last_clean":"2023-10-21T05:12:22.144525+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:12:22.144525+0000","last_undegraded":"2023-10-21T05:12:22.144525+0000","last_fullsized":"2023-10-21T05:12:22.144525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29515214867,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2089638,"num_read_kb":8248064,"num_write":2087560,"num_write_kb":8350234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17099915264,"data_stored":36703731750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40106,"ondisk_log_size":40106,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45150,"ondisk_log_size":45150,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739266,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10182400,"kb_used_data":8349304,"kb_used_omap":2268,"kb_used_meta":1830819,"kb_avail":84189440,"statfs":{"total":96636764160,"available":86209986560,"internally_reserved":0,"allocated":8549687296,"data_stored":18350630220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":1874759660},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.875,"15min":0.92700000000000005},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":2.4399999999999999,"5min":3.6960000000000002,"15min":4.649},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.78700000000000003,"15min":0.88800000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.3919999999999999,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.499}]},{"osd":1,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.91600000000000004,"15min":1.083},"min":{"1min":0.27500000000000002,"5min":0.26300000000000001,"15min":0.19900000000000001},"max":{"1min":2.2919999999999998,"5min":4.3879999999999999,"15min":27.196999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.98199999999999998,"15min":1.1950000000000001},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":2.3279999999999998,"5min":9.0020000000000007,"15min":27.242999999999999},"last":0.40000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739265,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5888148,"kb_used_data":4098636,"kb_used_omap":1038,"kb_used_meta":1788465,"kb_avail":88483692,"statfs":{"total":96636764160,"available":90607300608,"internally_reserved":0,"allocated":4197003264,"data_stored":9008112928,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1831388424},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.88,"15min":0.85199999999999998},"min":{"1min":0.20200000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":5.3920000000000003,"5min":6.0899999999999999,"15min":10.711},"last":2.5920000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.83199999999999996,"15min":0.94299999999999995},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":5.4569999999999999,"5min":6.0579999999999998,"15min":10.512},"last":2.6680000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:14:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.98499999999999999,"15min":1.0589999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":3.3239999999999998,"5min":4.4969999999999999,"15min":10.167999999999999},"last":2.6339999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.99199999999999999,"15min":1.109},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.5030000000000001,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.42099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6252232,"kb_used_data":4254720,"kb_used_omap":1073,"kb_used_meta":1996430,"kb_avail":88119608,"statfs":{"total":96636764160,"available":90234478592,"internally_reserved":0,"allocated":4356833280,"data_stored":9346289143,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1099032,"internal_metadata":2044345064},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.84699999999999998,"15min":0.84299999999999997},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.183},"max":{"1min":3.5369999999999999,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":1.109},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.90300000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.4889999999999999,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":0.254}]},{"osd":2,"last update":"Sat Oct 21 05:14:14 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.88600000000000001,"15min":0.94699999999999995},"min":{"1min":0.222,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":1.9219999999999999,"5min":4.359,"15min":5.3179999999999996},"last":0.218},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":1.0149999999999999},"min":{"1min":0.27300000000000002,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.165,"5min":5.133,"15min":5.133},"last":0.23599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4355100672,"data_stored":9345146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4195545088,"data_stored":9007407123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8549269504,"data_stored":18351177747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:18.701 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:18.701 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:18.702 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:18.702 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:18.703 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:38.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:39.241 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:14:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:14:39.263 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2388,"stamp":"2023-10-21T05:14:37.850637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29546764029,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2146902,"num_read_kb":8320202,"num_write":2096266,"num_write_kb":8382839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85620,"ondisk_log_size":85620,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21975148,"kb_used_data":16762964,"kb_used_omap":4658,"kb_used_meta":5207501,"kb_avail":261140372,"statfs":{"total":289910292480,"available":267407740928,"internally_reserved":0,"allocated":17165275136,"data_stored":36766783587,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4770566,"internal_metadata":5332481274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,7,3,12,21,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19738624,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4902,"num_read_kb":19528,"num_write":4807,"num_write_kb":19228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002043"},"pg_stats":[{"pgid":"2.7","version":"141'7019","reported_seq":10376,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.218101+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:14:33.218101+0000","last_peered":"2023-10-21T05:14:33.218101+0000","last_clean":"2023-10-21T05:14:33.218101+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:14:33.218101+0000","last_undegraded":"2023-10-21T05:14:33.218101+0000","last_fullsized":"2023-10-21T05:14:33.218101+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7019,"log_dups_size":0,"ondisk_log_size":7019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'3595","reported_seq":7310,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:31.632771+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:14:31.632771+0000","last_peered":"2023-10-21T05:14:31.632771+0000","last_clean":"2023-10-21T05:14:31.632771+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:14:31.632771+0000","last_undegraded":"2023-10-21T05:14:31.632771+0000","last_fullsized":"2023-10-21T05:14:31.632771+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3595,"log_dups_size":0,"ondisk_log_size":3595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'10471","reported_seq":15370,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.217059+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:14:33.217059+0000","last_peered":"2023-10-21T05:14:33.217059+0000","last_clean":"2023-10-21T05:14:33.217059+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:14:33.217059+0000","last_undegraded":"2023-10-21T05:14:33.217059+0000","last_fullsized":"2023-10-21T05:14:33.217059+0000","mapping_epoch":11,"log_start":"42'400","ondisk_log_start":"42'400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'7057","reported_seq":11428,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.216869+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:14:33.216869+0000","last_peered":"2023-10-21T05:14:33.216869+0000","last_clean":"2023-10-21T05:14:33.216869+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:14:33.216869+0000","last_undegraded":"2023-10-21T05:14:33.216869+0000","last_fullsized":"2023-10-21T05:14:33.216869+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7057,"log_dups_size":0,"ondisk_log_size":7057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'527510","reported_seq":1042565,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:37.632979+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:14:37.632979+0000","last_peered":"2023-10-21T05:14:37.632979+0000","last_clean":"2023-10-21T05:14:37.632979+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:14:37.632979+0000","last_undegraded":"2023-10-21T05:14:37.632979+0000","last_fullsized":"2023-10-21T05:14:37.632979+0000","mapping_epoch":14,"log_start":"140'517460","ondisk_log_start":"140'517460","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4562,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 499s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7308447744,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":519499,"num_read_kb":2050556,"num_write":518762,"num_write_kb":2075048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'14274","reported_seq":20952,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:37.408908+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:14:37.408908+0000","last_peered":"2023-10-21T05:14:37.408908+0000","last_clean":"2023-10-21T05:14:37.408908+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:14:37.408908+0000","last_undegraded":"2023-10-21T05:14:37.408908+0000","last_fullsized":"2023-10-21T05:14:37.408908+0000","mapping_epoch":11,"log_start":"78'4200","ondisk_log_start":"78'4200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'532579","reported_seq":1054052,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:33.237137+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:14:33.237137+0000","last_peered":"2023-10-21T05:14:33.237137+0000","last_clean":"2023-10-21T05:14:33.237137+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:14:33.237137+0000","last_undegraded":"2023-10-21T05:14:33.237137+0000","last_fullsized":"2023-10-21T05:14:33.237137+0000","mapping_epoch":14,"log_start":"140'522559","ondisk_log_start":"140'522559","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4057,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 478s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7396544512,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":526003,"num_read_kb":2075708,"num_write":524210,"num_write_kb":2096840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3491,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.218097+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:14:33.218097+0000","last_peered":"2023-10-21T05:14:33.218097+0000","last_clean":"2023-10-21T05:14:33.218097+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:14:33.218097+0000","last_undegraded":"2023-10-21T05:14:33.218097+0000","last_fullsized":"2023-10-21T05:14:33.218097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'556749","reported_seq":1098620,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:37.633279+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:14:37.633279+0000","last_peered":"2023-10-21T05:14:37.633279+0000","last_clean":"2023-10-21T05:14:37.633279+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:14:37.633279+0000","last_undegraded":"2023-10-21T05:14:37.633279+0000","last_fullsized":"2023-10-21T05:14:37.633279+0000","mapping_epoch":14,"log_start":"140'546650","ondisk_log_start":"140'546650","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3840,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 496s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7727775744,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":546646,"num_read_kb":2158052,"num_write":548314,"num_write_kb":2193256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6951,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.218684+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:14:33.218684+0000","last_peered":"2023-10-21T05:14:33.218684+0000","last_clean":"2023-10-21T05:14:33.218684+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:14:33.218684+0000","last_undegraded":"2023-10-21T05:14:33.218684+0000","last_fullsized":"2023-10-21T05:14:33.218684+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'508691","reported_seq":1009487,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.269710+0000","last_change":"2023-10-21T05:14:30.428801+0000","last_active":"2023-10-21T05:14:33.269710+0000","last_peered":"2023-10-21T05:14:33.269710+0000","last_clean":"2023-10-21T05:14:33.269710+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:14:33.269710+0000","last_undegraded":"2023-10-21T05:14:33.269710+0000","last_fullsized":"2023-10-21T05:14:33.269710+0000","mapping_epoch":14,"log_start":"140'498656","ondisk_log_start":"140'498656","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7113400339,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":505295,"num_read_kb":1994816,"num_write":503822,"num_write_kb":2015282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'6978","reported_seq":10266,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:33.217043+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:14:33.217043+0000","last_peered":"2023-10-21T05:14:33.217043+0000","last_clean":"2023-10-21T05:14:33.217043+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:14:33.217043+0000","last_undegraded":"2023-10-21T05:14:33.217043+0000","last_fullsized":"2023-10-21T05:14:33.217043+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":316,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:13:24.442385+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:13:24.442385+0000","last_peered":"2023-10-21T05:13:24.442385+0000","last_clean":"2023-10-21T05:13:24.442385+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:13:24.442385+0000","last_undegraded":"2023-10-21T05:13:24.442385+0000","last_fullsized":"2023-10-21T05:13:24.442385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29546168339,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2097443,"num_read_kb":8279132,"num_write":2095108,"num_write_kb":8380426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17161797632,"data_stored":36765614118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45317,"ondisk_log_size":45317,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739270,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10355876,"kb_used_data":8379804,"kb_used_omap":2268,"kb_used_meta":1973795,"kb_avail":84015964,"statfs":{"total":96636764160,"available":86032347136,"internally_reserved":0,"allocated":8580919296,"data_stored":18381862220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":2021167084},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.875,"15min":0.92700000000000005},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":2.4399999999999999,"5min":3.6960000000000002,"15min":4.649},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.78700000000000003,"15min":0.88800000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.3919999999999999,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.91600000000000004,"15min":1.083},"min":{"1min":0.27500000000000002,"5min":0.26300000000000001,"15min":0.19900000000000001},"max":{"1min":2.2919999999999998,"5min":4.3879999999999999,"15min":27.196999999999999},"last":0.218},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.98199999999999998,"15min":1.1950000000000001},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":2.3279999999999998,"5min":9.0020000000000007,"15min":27.242999999999999},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739269,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5992984,"kb_used_data":4113296,"kb_used_omap":1038,"kb_used_meta":1878641,"kb_avail":88378856,"statfs":{"total":96636764160,"available":90499948544,"internally_reserved":0,"allocated":4212015104,"data_stored":9023124768,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":1923728648},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,0,2,11,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.88,"15min":0.85199999999999998},"min":{"1min":0.20200000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":5.3920000000000003,"5min":6.0899999999999999,"15min":10.711},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.83199999999999996,"15min":0.94299999999999995},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":5.4569999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:14:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.98499999999999999,"15min":1.0589999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":3.3239999999999998,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.99199999999999999,"15min":1.109},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.5030000000000001,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5626288,"kb_used_data":4269864,"kb_used_omap":1351,"kb_used_meta":1355064,"kb_avail":88745552,"statfs":{"total":96636764160,"available":90875445248,"internally_reserved":0,"allocated":4372340736,"data_stored":9361796599,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1387585542},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,2,6,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.84699999999999998,"15min":0.84299999999999997},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.183},"max":{"1min":3.5369999999999999,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.90300000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.4889999999999999,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":0.308}]},{"osd":2,"last update":"Sat Oct 21 05:14:14 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.88600000000000001,"15min":0.94699999999999995},"min":{"1min":0.222,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":1.9219999999999999,"5min":4.359,"15min":5.3179999999999996},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":1.0149999999999999},"min":{"1min":0.27300000000000002,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.165,"5min":5.133,"15min":5.133},"last":0.35299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4370747392,"data_stored":9360793600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4210835456,"data_stored":9022697491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8580214784,"data_stored":18382123027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:39.264 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:39.265 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:39.265 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:39.266 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:39.266 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:39.539 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.535+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:14:39.540 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.535+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:14:39.540 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.535+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:14:39.540 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.535+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:14:39.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.535+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:14:39.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.535+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:14:39.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.536+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:14:39.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.536+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:14:39.542 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.536+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:14:39.542 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:14:39.536+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:14:53.590 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:14:53.586+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:14:59.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:59.803 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:14:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:14:59.829 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2399,"stamp":"2023-10-21T05:14:57.854620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29574252285,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2153700,"num_read_kb":8347226,"num_write":2102969,"num_write_kb":8409651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85589,"ondisk_log_size":85589,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22320168,"kb_used_data":16820620,"kb_used_omap":4658,"kb_used_meta":5494861,"kb_avail":260795352,"statfs":{"total":289910292480,"available":267054440448,"internally_reserved":0,"allocated":17224314880,"data_stored":36825800420,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4770566,"internal_metadata":5626737914},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,8,6,10,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15990784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3968,"num_read_kb":15756,"num_write":3896,"num_write_kb":15584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.7","version":"142'7047","reported_seq":10411,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:51.374389+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:14:51.374389+0000","last_peered":"2023-10-21T05:14:51.374389+0000","last_clean":"2023-10-21T05:14:51.374389+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:14:51.374389+0000","last_undegraded":"2023-10-21T05:14:51.374389+0000","last_fullsized":"2023-10-21T05:14:51.374389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'3609","reported_seq":7329,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:50.370290+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:14:50.370290+0000","last_peered":"2023-10-21T05:14:50.370290+0000","last_clean":"2023-10-21T05:14:50.370290+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:14:50.370290+0000","last_undegraded":"2023-10-21T05:14:50.370290+0000","last_fullsized":"2023-10-21T05:14:50.370290+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'10513","reported_seq":15422,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:51.379666+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:14:51.379666+0000","last_peered":"2023-10-21T05:14:51.379666+0000","last_clean":"2023-10-21T05:14:51.379666+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:14:51.379666+0000","last_undegraded":"2023-10-21T05:14:51.379666+0000","last_fullsized":"2023-10-21T05:14:51.379666+0000","mapping_epoch":11,"log_start":"44'500","ondisk_log_start":"44'500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'7085","reported_seq":11463,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:51.379529+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:14:51.379529+0000","last_peered":"2023-10-21T05:14:51.379529+0000","last_clean":"2023-10-21T05:14:51.379529+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:14:51.379529+0000","last_undegraded":"2023-10-21T05:14:51.379529+0000","last_fullsized":"2023-10-21T05:14:51.379529+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7085,"log_dups_size":0,"ondisk_log_size":7085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'529078","reported_seq":1045578,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:52.742415+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:14:52.742415+0000","last_peered":"2023-10-21T05:14:52.742415+0000","last_clean":"2023-10-21T05:14:52.742415+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:14:52.742415+0000","last_undegraded":"2023-10-21T05:14:52.742415+0000","last_fullsized":"2023-10-21T05:14:52.742415+0000","mapping_epoch":14,"log_start":"140'519060","ondisk_log_start":"140'519060","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4664,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 514s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7314518016,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":520935,"num_read_kb":2056260,"num_write":520243,"num_write_kb":2080972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'14298","reported_seq":20989,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:49.746772+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:14:49.746772+0000","last_peered":"2023-10-21T05:14:49.746772+0000","last_clean":"2023-10-21T05:14:49.746772+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:14:49.746772+0000","last_undegraded":"2023-10-21T05:14:49.746772+0000","last_fullsized":"2023-10-21T05:14:49.746772+0000","mapping_epoch":11,"log_start":"78'4200","ondisk_log_start":"78'4200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'534661","reported_seq":1058074,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:53.427802+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:14:53.427802+0000","last_peered":"2023-10-21T05:14:53.427802+0000","last_clean":"2023-10-21T05:14:53.427802+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:14:53.427802+0000","last_undegraded":"2023-10-21T05:14:53.427802+0000","last_fullsized":"2023-10-21T05:14:53.427802+0000","mapping_epoch":14,"log_start":"140'524660","ondisk_log_start":"140'524660","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4211,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 498s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7404568576,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":527928,"num_read_kb":2083360,"num_write":526167,"num_write_kb":2104668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3500,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:53.702478+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:14:53.702478+0000","last_peered":"2023-10-21T05:14:53.702478+0000","last_clean":"2023-10-21T05:14:53.702478+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:14:53.702478+0000","last_undegraded":"2023-10-21T05:14:53.702478+0000","last_fullsized":"2023-10-21T05:14:53.702478+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'558298","reported_seq":1101730,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:52.749256+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:14:52.749256+0000","last_peered":"2023-10-21T05:14:52.749256+0000","last_clean":"2023-10-21T05:14:52.749256+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:14:52.749256+0000","last_undegraded":"2023-10-21T05:14:52.749256+0000","last_fullsized":"2023-10-21T05:14:52.749256+0000","mapping_epoch":14,"log_start":"140'548250","ondisk_log_start":"140'548250","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3953,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 511s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7733764096,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":548195,"num_read_kb":2164220,"num_write":549772,"num_write_kb":2199088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6968,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:53.711338+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:14:53.711338+0000","last_peered":"2023-10-21T05:14:53.711338+0000","last_clean":"2023-10-21T05:14:53.711338+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:14:53.711338+0000","last_undegraded":"2023-10-21T05:14:53.711338+0000","last_fullsized":"2023-10-21T05:14:53.711338+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'510498","reported_seq":1013186,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:53.459972+0000","last_change":"2023-10-21T05:14:53.415729+0000","last_active":"2023-10-21T05:14:53.459972+0000","last_peered":"2023-10-21T05:14:53.459972+0000","last_clean":"2023-10-21T05:14:53.459972+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:14:53.459972+0000","last_undegraded":"2023-10-21T05:14:53.459972+0000","last_fullsized":"2023-10-21T05:14:53.459972+0000","mapping_epoch":14,"log_start":"140'500456","ondisk_log_start":"140'500456","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7120805907,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":507183,"num_read_kb":2002316,"num_write":505629,"num_write_kb":2022510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'7006","reported_seq":10301,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:51.383894+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:14:51.383894+0000","last_peered":"2023-10-21T05:14:51.383894+0000","last_clean":"2023-10-21T05:14:51.383894+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:14:51.383894+0000","last_undegraded":"2023-10-21T05:14:51.383894+0000","last_fullsized":"2023-10-21T05:14:51.383894+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7006,"log_dups_size":0,"ondisk_log_size":7006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":317,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:23.128718+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:14:23.128718+0000","last_peered":"2023-10-21T05:14:23.128718+0000","last_clean":"2023-10-21T05:14:23.128718+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:14:23.128718+0000","last_undegraded":"2023-10-21T05:14:23.128718+0000","last_fullsized":"2023-10-21T05:14:23.128718+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29573656595,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2104241,"num_read_kb":8306156,"num_write":2101811,"num_write_kb":8407238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17220644864,"data_stored":36824461350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40109,"ondisk_log_size":40109,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45381,"ondisk_log_size":45381,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739274,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10569328,"kb_used_data":8409956,"kb_used_omap":2268,"kb_used_meta":2157091,"kb_avail":83802512,"statfs":{"total":96636764160,"available":85813772288,"internally_reserved":0,"allocated":8611794944,"data_stored":18412730231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2322452,"internal_metadata":2208862188},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.875,"15min":0.92700000000000005},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":2.4399999999999999,"5min":3.6960000000000002,"15min":4.649},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.78700000000000003,"15min":0.88800000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.3919999999999999,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.66500000000000004}]},{"osd":1,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.91600000000000004,"15min":1.083},"min":{"1min":0.27500000000000002,"5min":0.26300000000000001,"15min":0.19900000000000001},"max":{"1min":2.2919999999999998,"5min":4.3879999999999999,"15min":27.196999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.98199999999999998,"15min":1.1950000000000001},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":2.3279999999999998,"5min":9.0020000000000007,"15min":27.242999999999999},"last":1.7090000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739273,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6098900,"kb_used_data":4129036,"kb_used_omap":1038,"kb_used_meta":1968817,"kb_avail":88272940,"statfs":{"total":96636764160,"available":90391490560,"internally_reserved":0,"allocated":4228132864,"data_stored":9039234891,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":2016068872},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:02 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.88,"15min":0.85199999999999998},"min":{"1min":0.20200000000000001,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":5.3920000000000003,"5min":6.0899999999999999,"15min":10.711},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.83199999999999996,"15min":0.94299999999999995},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":5.4569999999999999,"5min":6.0579999999999998,"15min":10.512},"last":0.308}]},{"osd":2,"last update":"Sat Oct 21 05:14:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.98499999999999999,"15min":1.0589999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":3.3239999999999998,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.99199999999999999,"15min":1.109},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.5030000000000001,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.53200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739273,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5651940,"kb_used_data":4281628,"kb_used_omap":1351,"kb_used_meta":1368952,"kb_avail":88719900,"statfs":{"total":96636764160,"available":90849177600,"internally_reserved":0,"allocated":4384387072,"data_stored":9373835298,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1401806854},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.84699999999999998,"15min":0.84299999999999997},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.183},"max":{"1min":3.5369999999999999,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.90300000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.4889999999999999,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:14:14 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.88600000000000001,"15min":0.94699999999999995},"min":{"1min":0.222,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":1.9219999999999999,"5min":4.359,"15min":5.3179999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":1.0149999999999999},"min":{"1min":0.27300000000000002,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.165,"5min":5.133,"15min":5.133},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4382789632,"data_stored":9372835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4226240512,"data_stored":9038102547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8611614720,"data_stored":18413522963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:59.831 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:59.832 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:59.833 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:59.834 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:59.835 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:19.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:15:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:15:20.412 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2410,"stamp":"2023-10-21T05:15:19.861584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29612910333,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2163559,"num_read_kb":8386502,"num_write":2112409,"num_write_kb":8447411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85734,"ondisk_log_size":85734,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22594996,"kb_used_data":16895388,"kb_used_omap":4878,"kb_used_meta":5694705,"kb_avail":260520524,"statfs":{"total":289910292480,"available":266773016576,"internally_reserved":0,"allocated":17300877312,"data_stored":36902362852,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4995685,"internal_metadata":5831378331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,7,9,2,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22974464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5823,"num_read_kb":23196,"num_write":5608,"num_write_kb":22432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002047"},"pg_stats":[{"pgid":"2.7","version":"142'7069","reported_seq":10443,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:16.614997+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:15:16.614997+0000","last_peered":"2023-10-21T05:15:16.614997+0000","last_clean":"2023-10-21T05:15:16.614997+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:15:16.614997+0000","last_undegraded":"2023-10-21T05:15:16.614997+0000","last_fullsized":"2023-10-21T05:15:16.614997+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'3620","reported_seq":7345,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:15.019359+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:15:15.019359+0000","last_peered":"2023-10-21T05:15:15.019359+0000","last_clean":"2023-10-21T05:15:15.019359+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:15:15.019359+0000","last_undegraded":"2023-10-21T05:15:15.019359+0000","last_fullsized":"2023-10-21T05:15:15.019359+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3620,"log_dups_size":0,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'10546","reported_seq":15470,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:16.619395+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:15:16.619395+0000","last_peered":"2023-10-21T05:15:16.619395+0000","last_clean":"2023-10-21T05:15:16.619395+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:15:16.619395+0000","last_undegraded":"2023-10-21T05:15:16.619395+0000","last_fullsized":"2023-10-21T05:15:16.619395+0000","mapping_epoch":11,"log_start":"44'500","ondisk_log_start":"44'500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'7107","reported_seq":11495,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:16.603103+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:15:16.603103+0000","last_peered":"2023-10-21T05:15:16.603103+0000","last_clean":"2023-10-21T05:15:16.603103+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:15:16.603103+0000","last_undegraded":"2023-10-21T05:15:16.603103+0000","last_fullsized":"2023-10-21T05:15:16.603103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'531613","reported_seq":1050559,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:18.009009+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:15:18.009009+0000","last_peered":"2023-10-21T05:15:18.009009+0000","last_clean":"2023-10-21T05:15:18.009009+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:15:18.009009+0000","last_undegraded":"2023-10-21T05:15:18.009009+0000","last_fullsized":"2023-10-21T05:15:18.009009+0000","mapping_epoch":14,"log_start":"141'521560","ondisk_log_start":"141'521560","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4931,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 540s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7323979776,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":523373,"num_read_kb":2065988,"num_write":522558,"num_write_kb":2090232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'14386","reported_seq":21097,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:15.626830+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:15:15.626830+0000","last_peered":"2023-10-21T05:15:15.626830+0000","last_clean":"2023-10-21T05:15:15.626830+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:15:15.626830+0000","last_undegraded":"2023-10-21T05:15:15.626830+0000","last_fullsized":"2023-10-21T05:15:15.626830+0000","mapping_epoch":11,"log_start":"78'4300","ondisk_log_start":"78'4300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'537232","reported_seq":1063135,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:18.552387+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:15:18.552387+0000","last_peered":"2023-10-21T05:15:18.552387+0000","last_clean":"2023-10-21T05:15:18.552387+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:15:18.552387+0000","last_undegraded":"2023-10-21T05:15:18.552387+0000","last_fullsized":"2023-10-21T05:15:18.552387+0000","mapping_epoch":14,"log_start":"141'527160","ondisk_log_start":"141'527160","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4467,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 523s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7414235136,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":530401,"num_read_kb":2093188,"num_write":528526,"num_write_kb":2114104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3511,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:19.297596+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:15:19.297596+0000","last_peered":"2023-10-21T05:15:19.297596+0000","last_clean":"2023-10-21T05:15:19.297596+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:15:19.297596+0000","last_undegraded":"2023-10-21T05:15:19.297596+0000","last_fullsized":"2023-10-21T05:15:19.297596+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'560978","reported_seq":1107044,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:18.010247+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:15:18.010247+0000","last_peered":"2023-10-21T05:15:18.010247+0000","last_clean":"2023-10-21T05:15:18.010247+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:15:18.010247+0000","last_undegraded":"2023-10-21T05:15:18.010247+0000","last_fullsized":"2023-10-21T05:15:18.010247+0000","mapping_epoch":14,"log_start":"141'550950","ondisk_log_start":"141'550950","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4170,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 537s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7744036864,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":550819,"num_read_kb":2174668,"num_write":552277,"num_write_kb":2209108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":6988,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:19.299261+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:15:19.299261+0000","last_peered":"2023-10-21T05:15:19.299261+0000","last_clean":"2023-10-21T05:15:19.299261+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:15:19.299261+0000","last_undegraded":"2023-10-21T05:15:19.299261+0000","last_fullsized":"2023-10-21T05:15:19.299261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'512759","reported_seq":1017778,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:18.552497+0000","last_change":"2023-10-21T05:15:14.622477+0000","last_active":"2023-10-21T05:15:18.552497+0000","last_peered":"2023-10-21T05:15:18.552497+0000","last_clean":"2023-10-21T05:15:18.552497+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:15:18.552497+0000","last_undegraded":"2023-10-21T05:15:18.552497+0000","last_fullsized":"2023-10-21T05:15:18.552497+0000","mapping_epoch":14,"log_start":"141'502756","ondisk_log_start":"141'502756","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7130062867,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":509507,"num_read_kb":2011588,"num_write":507890,"num_write_kb":2031554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'7028","reported_seq":10334,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:16.630684+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:15:16.630684+0000","last_peered":"2023-10-21T05:15:16.630684+0000","last_clean":"2023-10-21T05:15:16.630684+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:15:16.630684+0000","last_undegraded":"2023-10-21T05:15:16.630684+0000","last_fullsized":"2023-10-21T05:15:16.630684+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7028,"log_dups_size":0,"ondisk_log_size":7028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":317,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:23.128718+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:14:23.128718+0000","last_peered":"2023-10-21T05:14:23.128718+0000","last_clean":"2023-10-21T05:14:23.128718+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:14:23.128718+0000","last_undegraded":"2023-10-21T05:14:23.128718+0000","last_fullsized":"2023-10-21T05:14:23.128718+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29612314643,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2114100,"num_read_kb":8345432,"num_write":2111251,"num_write_kb":8444998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17298014208,"data_stored":36901830694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40156,"ondisk_log_size":40156,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45479,"ondisk_log_size":45479,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739279,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10606980,"kb_used_data":8447804,"kb_used_omap":2487,"kb_used_meta":2156680,"kb_avail":83764860,"statfs":{"total":96636764160,"available":85775216640,"internally_reserved":0,"allocated":8650551296,"data_stored":18451486583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2547571,"internal_metadata":2208440461},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":4.3090000000000002,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.79800000000000004,"15min":0.874},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.2030000000000003,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.96199999999999997,"15min":1.107},"min":{"1min":0.218,"5min":0.218,"15min":0.19900000000000001},"max":{"1min":4.2789999999999999,"5min":4.3879999999999999,"15min":27.196999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.997,"15min":1.1910000000000001},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":4.2510000000000003,"5min":9.0020000000000007,"15min":27.242999999999999},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359739278,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6222624,"kb_used_data":4146840,"kb_used_omap":1038,"kb_used_meta":2074737,"kb_avail":88149216,"statfs":{"total":96636764160,"available":90264797184,"internally_reserved":0,"allocated":4246364160,"data_stored":9057466187,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1063672,"internal_metadata":2124530952},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,3,2,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.80200000000000005,"15min":0.81999999999999995},"min":{"1min":0.20399999999999999,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":2.5920000000000001,"5min":6.0899999999999999,"15min":10.711},"last":0.315},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.92300000000000004},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":2.6680000000000001,"5min":6.0579999999999998,"15min":10.512},"last":0.35399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.95499999999999996,"15min":1.0569999999999999},"min":{"1min":0.245,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":2.6339999999999999,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.93200000000000005,"15min":1.093},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.5470000000000002,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739278,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5765392,"kb_used_data":4300744,"kb_used_omap":1351,"kb_used_meta":1463288,"kb_avail":88606448,"statfs":{"total":96636764160,"available":90733002752,"internally_reserved":0,"allocated":4403961856,"data_stored":9393410082,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1498406918},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.85299999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.183},"max":{"1min":2.6419999999999999,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86899999999999999,"15min":0.91000000000000003},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.4050000000000002,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.71199999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:15:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88,"15min":0.95399999999999996},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5840000000000001,"5min":4.359,"15min":5.3179999999999996},"last":0.38},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.84699999999999998,"15min":0.98399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.0510000000000002,"5min":4.3049999999999997,"15min":5.133},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4402524160,"data_stored":9392570368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4245155840,"data_stored":9057017875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8650334208,"data_stored":18452242451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:20.414 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:20.415 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:20.415 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:20.416 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:20.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:15:20.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T05:15:20.813+0000 7fca7b7fe640 0 --2- 172.21.15.40:0/3302682727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7c08b700 0x7fca7c1bde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T05:15:21.155 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:15:21.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:15:21.718 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:15:21.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:15:22.261 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:15:22.284 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:15:22.812 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:15:22.836 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:29.176 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.167+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:15:29.177 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.172+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:15:29.178 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.172+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:15:29.178 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.172+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:15:29.187 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.184+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:15:29.187 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.184+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:15:29.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.184+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:15:29.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:29.184+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:15:31.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:15:31.612 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:15:31.613 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:15:31.613 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:15:31.613 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:15:31.614 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:15:31.614 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:15:31.614 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:15:31.615 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:15:31.615 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:15:31.609+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:15:33.160 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-21T05:15:33.157+0000 7f4e3e65e640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-21T05:15:33.200 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:15:33.197+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:15:33.201 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:15:33.197+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:15:33.201 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:15:33.197+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:15:33.201 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:15:33.197+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:15:42.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:43.371 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:15:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:15:43.390 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2421,"stamp":"2023-10-21T05:15:41.865471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29645141757,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2171545,"num_read_kb":8418258,"num_write":2120271,"num_write_kb":8478859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85776,"ondisk_log_size":85776,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20942636,"kb_used_data":16958804,"kb_used_omap":5031,"kb_used_meta":3978776,"kb_avail":262172884,"statfs":{"total":289910292480,"available":268465033216,"internally_reserved":0,"allocated":17365815296,"data_stored":36967300836,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":4074266693},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,6,7,13,17,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16355328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4078,"num_read_kb":16188,"num_write":3988,"num_write_kb":15952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002065"},"pg_stats":[{"pgid":"2.7","version":"142'7089","reported_seq":10471,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:36.251872+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:15:36.251872+0000","last_peered":"2023-10-21T05:15:36.251872+0000","last_clean":"2023-10-21T05:15:36.251872+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:15:36.251872+0000","last_undegraded":"2023-10-21T05:15:36.251872+0000","last_fullsized":"2023-10-21T05:15:36.251872+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'3630","reported_seq":7359,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:34.657282+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:15:34.657282+0000","last_peered":"2023-10-21T05:15:34.657282+0000","last_clean":"2023-10-21T05:15:34.657282+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:15:34.657282+0000","last_undegraded":"2023-10-21T05:15:34.657282+0000","last_fullsized":"2023-10-21T05:15:34.657282+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'10576","reported_seq":15513,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:36.263180+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:15:36.263180+0000","last_peered":"2023-10-21T05:15:36.263180+0000","last_clean":"2023-10-21T05:15:36.263180+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:15:36.263180+0000","last_undegraded":"2023-10-21T05:15:36.263180+0000","last_fullsized":"2023-10-21T05:15:36.263180+0000","mapping_epoch":11,"log_start":"44'500","ondisk_log_start":"44'500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'7127","reported_seq":11524,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:36.251859+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:15:36.251859+0000","last_peered":"2023-10-21T05:15:36.251859+0000","last_clean":"2023-10-21T05:15:36.251859+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:15:36.251859+0000","last_undegraded":"2023-10-21T05:15:36.251859+0000","last_fullsized":"2023-10-21T05:15:36.251859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7127,"log_dups_size":0,"ondisk_log_size":7127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'533666","reported_seq":1054623,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:38.328043+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:15:38.328043+0000","last_peered":"2023-10-21T05:15:38.328043+0000","last_clean":"2023-10-21T05:15:38.328043+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:15:38.328043+0000","last_undegraded":"2023-10-21T05:15:38.328043+0000","last_fullsized":"2023-10-21T05:15:38.328043+0000","mapping_epoch":14,"log_start":"141'523660","ondisk_log_start":"141'523660","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5114,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 560s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7331803136,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":525363,"num_read_kb":2073916,"num_write":524460,"num_write_kb":2097840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'14430","reported_seq":21158,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:36.806771+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:15:36.806771+0000","last_peered":"2023-10-21T05:15:36.806771+0000","last_clean":"2023-10-21T05:15:36.806771+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:15:36.806771+0000","last_undegraded":"2023-10-21T05:15:36.806771+0000","last_fullsized":"2023-10-21T05:15:36.806771+0000","mapping_epoch":11,"log_start":"80'4400","ondisk_log_start":"80'4400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'539279","reported_seq":1067200,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:38.675443+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:15:38.675443+0000","last_peered":"2023-10-21T05:15:38.675443+0000","last_clean":"2023-10-21T05:15:38.675443+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:15:38.675443+0000","last_undegraded":"2023-10-21T05:15:38.675443+0000","last_fullsized":"2023-10-21T05:15:38.675443+0000","mapping_epoch":14,"log_start":"141'529260","ondisk_log_start":"141'529260","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4590,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 543s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7422238720,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":532405,"num_read_kb":2101148,"num_write":530472,"num_write_kb":2121888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3519,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:39.836620+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:15:39.836620+0000","last_peered":"2023-10-21T05:15:39.836620+0000","last_clean":"2023-10-21T05:15:39.836620+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:15:39.836620+0000","last_undegraded":"2023-10-21T05:15:39.836620+0000","last_fullsized":"2023-10-21T05:15:39.836620+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'563136","reported_seq":1111288,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:38.297364+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:15:38.297364+0000","last_peered":"2023-10-21T05:15:38.297364+0000","last_clean":"2023-10-21T05:15:38.297364+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:15:38.297364+0000","last_undegraded":"2023-10-21T05:15:38.297364+0000","last_fullsized":"2023-10-21T05:15:38.297364+0000","mapping_epoch":14,"log_start":"141'553051","ondisk_log_start":"141'553051","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4274,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 557s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7752507392,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":552887,"num_read_kb":2182904,"num_write":554350,"num_write_kb":2217400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7004,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:39.789432+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:15:39.789432+0000","last_peered":"2023-10-21T05:15:39.789432+0000","last_clean":"2023-10-21T05:15:39.789432+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:15:39.789432+0000","last_undegraded":"2023-10-21T05:15:39.789432+0000","last_fullsized":"2023-10-21T05:15:39.789432+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'514700","reported_seq":1021654,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:38.688165+0000","last_change":"2023-10-21T05:15:35.747181+0000","last_active":"2023-10-21T05:15:38.688165+0000","last_peered":"2023-10-21T05:15:38.688165+0000","last_clean":"2023-10-21T05:15:38.688165+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:15:38.688165+0000","last_undegraded":"2023-10-21T05:15:38.688165+0000","last_fullsized":"2023-10-21T05:15:38.688165+0000","mapping_epoch":14,"log_start":"141'504656","ondisk_log_start":"141'504656","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7137996819,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":511431,"num_read_kb":2019220,"num_write":509831,"num_write_kb":2039318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'7048","reported_seq":10362,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:15:36.262486+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:15:36.262486+0000","last_peered":"2023-10-21T05:15:36.262486+0000","last_clean":"2023-10-21T05:15:36.262486+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:15:36.262486+0000","last_undegraded":"2023-10-21T05:15:36.262486+0000","last_fullsized":"2023-10-21T05:15:36.262486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":317,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-21T05:14:23.128718+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:14:23.128718+0000","last_peered":"2023-10-21T05:14:23.128718+0000","last_clean":"2023-10-21T05:14:23.128718+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:14:23.128718+0000","last_undegraded":"2023-10-21T05:14:23.128718+0000","last_fullsized":"2023-10-21T05:14:23.128718+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29644546067,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2122086,"num_read_kb":8377188,"num_write":2119113,"num_write_kb":8476446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17362153472,"data_stored":36965969958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40154,"ondisk_log_size":40154,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45523,"ondisk_log_size":45523,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9369216,"kb_used_data":8478968,"kb_used_omap":2489,"kb_used_meta":887750,"kb_avail":85002624,"statfs":{"total":96636764160,"available":87042686976,"internally_reserved":0,"allocated":8682463232,"data_stored":18483398519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":909056509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":4.3090000000000002,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.79800000000000004,"15min":0.874},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.2030000000000003,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":1.196}]},{"osd":1,"last update":"Sat Oct 21 05:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.96199999999999997,"15min":1.107},"min":{"1min":0.218,"5min":0.218,"15min":0.19900000000000001},"max":{"1min":4.2789999999999999,"5min":4.3879999999999999,"15min":27.196999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.997,"15min":1.1910000000000001},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":4.2510000000000003,"5min":9.0020000000000007,"15min":27.242999999999999},"last":0.995}]}]},{"osd":1,"up_from":8,"seq":34359739282,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5715692,"kb_used_data":4163108,"kb_used_omap":1190,"kb_used_meta":1551385,"kb_avail":88656148,"statfs":{"total":96636764160,"available":90783895552,"internally_reserved":0,"allocated":4263022592,"data_stored":9074124619,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":1588618818},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,4,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.80200000000000005,"15min":0.81999999999999995},"min":{"1min":0.20399999999999999,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":2.5920000000000001,"5min":6.0899999999999999,"15min":10.711},"last":0.378},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.92300000000000004},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":2.6680000000000001,"5min":6.0579999999999998,"15min":10.512},"last":0.437}]},{"osd":2,"last update":"Sat Oct 21 05:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.95499999999999996,"15min":1.0569999999999999},"min":{"1min":0.245,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":2.6339999999999999,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.93200000000000005,"15min":1.093},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.5470000000000002,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.48399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5857728,"kb_used_data":4316728,"kb_used_omap":1351,"kb_used_meta":1539640,"kb_avail":88514112,"statfs":{"total":96636764160,"available":90638450688,"internally_reserved":0,"allocated":4420329472,"data_stored":9409777698,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1576591366},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,0,8,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.85299999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.183},"max":{"1min":2.6419999999999999,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86899999999999999,"15min":0.91000000000000003},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.4050000000000002,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":2.5129999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:15:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88,"15min":0.95399999999999996},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5840000000000001,"5min":4.359,"15min":5.3179999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.84699999999999998,"15min":0.98399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.0510000000000002,"5min":4.3049999999999997,"15min":5.133},"last":0.29999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4418822144,"data_stored":9408868352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4261089280,"data_stored":9072951315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8682242048,"data_stored":18484150291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:43.391 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:43.392 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:43.392 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:43.393 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:43.393 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:03.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:03.711 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:16:03.710+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:16:03.712 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:16:03.710+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:16:03.712 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:16:03.710+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:16:03.712 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:16:03.710+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:16:03.923 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:16:03.924 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:16:03.948 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2433,"stamp":"2023-10-21T05:16:03.872065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29681219325,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2180592,"num_read_kb":8454210,"num_write":2129079,"num_write_kb":8514091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85872,"ondisk_log_size":85872,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21251056,"kb_used_data":17025108,"kb_used_omap":5031,"kb_used_meta":4220888,"kb_avail":261864464,"statfs":{"total":289910292480,"available":268149211136,"internally_reserved":0,"allocated":17433710592,"data_stored":37035173092,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":4322189381},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,12,2,18,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11612160,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3000,"num_read_kb":11924,"num_write":2839,"num_write_kb":11356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001719"},"pg_stats":[{"pgid":"2.7","version":"143'7111","reported_seq":10502,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:56.290836+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:15:56.290836+0000","last_peered":"2023-10-21T05:15:56.290836+0000","last_clean":"2023-10-21T05:15:56.290836+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:15:56.290836+0000","last_undegraded":"2023-10-21T05:15:56.290836+0000","last_fullsized":"2023-10-21T05:15:56.290836+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'3641","reported_seq":7375,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:56.052975+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:15:56.052975+0000","last_peered":"2023-10-21T05:15:56.052975+0000","last_clean":"2023-10-21T05:15:56.052975+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:15:56.052975+0000","last_undegraded":"2023-10-21T05:15:56.052975+0000","last_fullsized":"2023-10-21T05:15:56.052975+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'10609","reported_seq":15559,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:56.303065+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:15:56.303065+0000","last_peered":"2023-10-21T05:15:56.303065+0000","last_clean":"2023-10-21T05:15:56.303065+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:15:56.303065+0000","last_undegraded":"2023-10-21T05:15:56.303065+0000","last_fullsized":"2023-10-21T05:15:56.303065+0000","mapping_epoch":11,"log_start":"47'600","ondisk_log_start":"47'600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'7149","reported_seq":11555,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:56.287575+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:15:56.287575+0000","last_peered":"2023-10-21T05:15:56.287575+0000","last_clean":"2023-10-21T05:15:56.287575+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:15:56.287575+0000","last_undegraded":"2023-10-21T05:15:56.287575+0000","last_fullsized":"2023-10-21T05:15:56.287575+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'536144","reported_seq":1059587,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:03.661694+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:16:03.661694+0000","last_peered":"2023-10-21T05:16:03.661694+0000","last_clean":"2023-10-21T05:16:03.661694+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:03.661694+0000","last_undegraded":"2023-10-21T05:16:03.661694+0000","last_fullsized":"2023-10-21T05:16:03.661694+0000","mapping_epoch":14,"log_start":"141'526061","ondisk_log_start":"141'526061","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5302,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 585s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7341326336,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":527830,"num_read_kb":2083720,"num_write":526781,"num_write_kb":2107124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'14506","reported_seq":21255,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:02.712619+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:16:02.712619+0000","last_peered":"2023-10-21T05:16:02.712619+0000","last_clean":"2023-10-21T05:16:02.712619+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:16:02.712619+0000","last_undegraded":"2023-10-21T05:16:02.712619+0000","last_fullsized":"2023-10-21T05:16:02.712619+0000","mapping_epoch":11,"log_start":"80'4500","ondisk_log_start":"80'4500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'541427","reported_seq":1071376,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:58.854180+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:15:58.854180+0000","last_peered":"2023-10-21T05:15:58.854180+0000","last_clean":"2023-10-21T05:15:58.854180+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:15:58.854180+0000","last_undegraded":"2023-10-21T05:15:58.854180+0000","last_fullsized":"2023-10-21T05:15:58.854180+0000","mapping_epoch":14,"log_start":"141'531360","ondisk_log_start":"141'531360","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4754,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 563s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7430479872,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":534426,"num_read_kb":2109164,"num_write":532487,"num_write_kb":2129948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3526,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:55.143360+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:15:55.143360+0000","last_peered":"2023-10-21T05:15:55.143360+0000","last_clean":"2023-10-21T05:15:55.143360+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:15:55.143360+0000","last_undegraded":"2023-10-21T05:15:55.143360+0000","last_fullsized":"2023-10-21T05:15:55.143360+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'565818","reported_seq":1116616,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:03.621125+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:16:03.621125+0000","last_peered":"2023-10-21T05:16:03.621125+0000","last_clean":"2023-10-21T05:16:03.621125+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:03.621125+0000","last_undegraded":"2023-10-21T05:16:03.621125+0000","last_fullsized":"2023-10-21T05:16:03.621125+0000","mapping_epoch":14,"log_start":"141'555752","ondisk_log_start":"141'555752","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4413,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 582s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7763025920,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":555521,"num_read_kb":2193360,"num_write":556918,"num_write_kb":2227672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7017,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:55.137406+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:15:55.137406+0000","last_peered":"2023-10-21T05:15:55.137406+0000","last_clean":"2023-10-21T05:15:55.137406+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:15:55.137406+0000","last_undegraded":"2023-10-21T05:15:55.137406+0000","last_fullsized":"2023-10-21T05:15:55.137406+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'516604","reported_seq":1025494,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:58.847706+0000","last_change":"2023-10-21T05:15:57.786967+0000","last_active":"2023-10-21T05:15:58.847706+0000","last_peered":"2023-10-21T05:15:58.847706+0000","last_clean":"2023-10-21T05:15:58.847706+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:15:58.847706+0000","last_undegraded":"2023-10-21T05:15:58.847706+0000","last_fullsized":"2023-10-21T05:15:58.847706+0000","mapping_epoch":14,"log_start":"141'506556","ondisk_log_start":"141'506556","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7145791507,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":513356,"num_read_kb":2026896,"num_write":511735,"num_write_kb":2046934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'7070","reported_seq":10393,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:56.362903+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:15:56.362903+0000","last_peered":"2023-10-21T05:15:56.362903+0000","last_clean":"2023-10-21T05:15:56.362903+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:15:56.362903+0000","last_undegraded":"2023-10-21T05:15:56.362903+0000","last_fullsized":"2023-10-21T05:15:56.362903+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7070,"log_dups_size":0,"ondisk_log_size":7070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":318,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:44.469051+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:14:44.469051+0000","last_peered":"2023-10-21T05:14:44.469051+0000","last_clean":"2023-10-21T05:14:44.469051+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:14:44.469051+0000","last_undegraded":"2023-10-21T05:14:44.469051+0000","last_fullsized":"2023-10-21T05:14:44.469051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29680623635,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2131133,"num_read_kb":8413140,"num_write":2127921,"num_write_kb":8511678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17430003712,"data_stored":37033820198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45509,"ondisk_log_size":45509,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739287,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9449080,"kb_used_data":8510256,"kb_used_omap":2489,"kb_used_meta":936326,"kb_avail":84922760,"statfs":{"total":96636764160,"available":86960906240,"internally_reserved":0,"allocated":8714502144,"data_stored":18515429751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":958798333},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":4.3090000000000002,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.79800000000000004,"15min":0.874},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.2030000000000003,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":4.2519999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:15:11 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.96199999999999997,"15min":1.107},"min":{"1min":0.218,"5min":0.218,"15min":0.19900000000000001},"max":{"1min":4.2789999999999999,"5min":4.3879999999999999,"15min":27.196999999999999},"last":1.964},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.997,"15min":1.1910000000000001},"min":{"1min":0.25600000000000001,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":4.2510000000000003,"5min":9.0020000000000007,"15min":27.242999999999999},"last":3.2669999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739286,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5829096,"kb_used_data":4178720,"kb_used_omap":1190,"kb_used_meta":1649177,"kb_avail":88542744,"statfs":{"total":96636764160,"available":90667769856,"internally_reserved":0,"allocated":4279009280,"data_stored":9090103627,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":1688757826},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.80200000000000005,"15min":0.81999999999999995},"min":{"1min":0.20399999999999999,"5min":0.16500000000000001,"15min":0.16400000000000001},"max":{"1min":2.5920000000000001,"5min":6.0899999999999999,"15min":10.711},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.92300000000000004},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":2.6680000000000001,"5min":6.0579999999999998,"15min":10.512},"last":0.379}]},{"osd":2,"last update":"Sat Oct 21 05:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.95499999999999996,"15min":1.0569999999999999},"min":{"1min":0.245,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":2.6339999999999999,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.93200000000000005,"15min":1.093},"min":{"1min":0.24399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.5470000000000002,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":1.9390000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5972880,"kb_used_data":4336132,"kb_used_omap":1351,"kb_used_meta":1635384,"kb_avail":88398960,"statfs":{"total":96636764160,"available":90520535040,"internally_reserved":0,"allocated":4440199168,"data_stored":9429639714,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1674633222},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,12,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.85299999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.183},"max":{"1min":2.6419999999999999,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86899999999999999,"15min":0.91000000000000003},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.4050000000000002,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.82099999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:15:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.88,"15min":0.95399999999999996},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5840000000000001,"5min":4.359,"15min":5.3179999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.84699999999999998,"15min":0.98399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":2.0510000000000002,"5min":4.3049999999999997,"15min":5.133},"last":0.89000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4438863872,"data_stored":9428910080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4277129216,"data_stored":9088991251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8714010624,"data_stored":18515918867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:03.950 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:03.950 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:03.951 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:03.951 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:03.951 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:23.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:24.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:16:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:16:24.512 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2443,"stamp":"2023-10-21T05:16:23.876808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29707331325,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2187402,"num_read_kb":8481302,"num_write":2135442,"num_write_kb":8539543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85976,"ondisk_log_size":85976,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21591616,"kb_used_data":17079840,"kb_used_omap":5031,"kb_used_meta":4506712,"kb_avail":261523904,"statfs":{"total":289910292480,"available":267800477696,"internally_reserved":0,"allocated":17489756160,"data_stored":37091218660,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":4614873157},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,14,20,7,21,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14602240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3825,"num_read_kb":15204,"num_write":3557,"num_write_kb":14228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"2.7","version":"143'7137","reported_seq":10537,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.095046+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:16:16.095046+0000","last_peered":"2023-10-21T05:16:16.095046+0000","last_clean":"2023-10-21T05:16:16.095046+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:16:16.095046+0000","last_undegraded":"2023-10-21T05:16:16.095046+0000","last_fullsized":"2023-10-21T05:16:16.095046+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'3654","reported_seq":7392,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.617753+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:16:15.617753+0000","last_peered":"2023-10-21T05:16:15.617753+0000","last_clean":"2023-10-21T05:16:15.617753+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:16:15.617753+0000","last_undegraded":"2023-10-21T05:16:15.617753+0000","last_fullsized":"2023-10-21T05:16:15.617753+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3654,"log_dups_size":0,"ondisk_log_size":3654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'10648","reported_seq":15610,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.102238+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:16:16.102238+0000","last_peered":"2023-10-21T05:16:16.102238+0000","last_clean":"2023-10-21T05:16:16.102238+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:16:16.102238+0000","last_undegraded":"2023-10-21T05:16:16.102238+0000","last_fullsized":"2023-10-21T05:16:16.102238+0000","mapping_epoch":11,"log_start":"47'600","ondisk_log_start":"47'600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'7175","reported_seq":11589,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.085932+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:16:16.085932+0000","last_peered":"2023-10-21T05:16:16.085932+0000","last_clean":"2023-10-21T05:16:16.085932+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:16:16.085932+0000","last_undegraded":"2023-10-21T05:16:16.085932+0000","last_fullsized":"2023-10-21T05:16:16.085932+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7175,"log_dups_size":0,"ondisk_log_size":7175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'537674","reported_seq":1062574,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:18.824728+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:16:18.824728+0000","last_peered":"2023-10-21T05:16:18.824728+0000","last_clean":"2023-10-21T05:16:18.824728+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:18.824728+0000","last_undegraded":"2023-10-21T05:16:18.824728+0000","last_fullsized":"2023-10-21T05:16:18.824728+0000","mapping_epoch":14,"log_start":"141'527661","ondisk_log_start":"141'527661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5502,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 600s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7346962432,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":529273,"num_read_kb":2089448,"num_write":528146,"num_write_kb":2112584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'14538","reported_seq":21299,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:18.718310+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:16:18.718310+0000","last_peered":"2023-10-21T05:16:18.718310+0000","last_clean":"2023-10-21T05:16:18.718310+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:16:18.718310+0000","last_undegraded":"2023-10-21T05:16:18.718310+0000","last_fullsized":"2023-10-21T05:16:18.718310+0000","mapping_epoch":11,"log_start":"80'4500","ondisk_log_start":"80'4500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'543454","reported_seq":1075402,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:18.973260+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:16:18.973260+0000","last_peered":"2023-10-21T05:16:18.973260+0000","last_clean":"2023-10-21T05:16:18.973260+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:16:18.973260+0000","last_undegraded":"2023-10-21T05:16:18.973260+0000","last_fullsized":"2023-10-21T05:16:18.973260+0000","mapping_epoch":14,"log_start":"141'533360","ondisk_log_start":"141'533360","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4979,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 583s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7438049280,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":536406,"num_read_kb":2117032,"num_write":534333,"num_write_kb":2137332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3534,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.620108+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:16:15.620108+0000","last_peered":"2023-10-21T05:16:15.620108+0000","last_clean":"2023-10-21T05:16:15.620108+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:16:15.620108+0000","last_undegraded":"2023-10-21T05:16:15.620108+0000","last_fullsized":"2023-10-21T05:16:15.620108+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'567374","reported_seq":1119721,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:18.812354+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:16:18.812354+0000","last_peered":"2023-10-21T05:16:18.812354+0000","last_clean":"2023-10-21T05:16:18.812354+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:18.812354+0000","last_undegraded":"2023-10-21T05:16:18.812354+0000","last_fullsized":"2023-10-21T05:16:18.812354+0000","mapping_epoch":14,"log_start":"141'557352","ondisk_log_start":"141'557352","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4573,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 597s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7768858624,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":557058,"num_read_kb":2199472,"num_write":558341,"num_write_kb":2233364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7034,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.635908+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:16:15.635908+0000","last_peered":"2023-10-21T05:16:15.635908+0000","last_clean":"2023-10-21T05:16:15.635908+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:16:15.635908+0000","last_undegraded":"2023-10-21T05:16:15.635908+0000","last_fullsized":"2023-10-21T05:16:15.635908+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'518333","reported_seq":1029078,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:18.997907+0000","last_change":"2023-10-21T05:16:18.838476+0000","last_active":"2023-10-21T05:16:18.997907+0000","last_peered":"2023-10-21T05:16:18.997907+0000","last_clean":"2023-10-21T05:16:18.997907+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:16:18.997907+0000","last_undegraded":"2023-10-21T05:16:18.997907+0000","last_fullsized":"2023-10-21T05:16:18.997907+0000","mapping_epoch":14,"log_start":"141'508256","ondisk_log_start":"141'508256","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7152865299,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":515206,"num_read_kb":2034280,"num_write":513464,"num_write_kb":2053850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'7096","reported_seq":10427,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:16.100451+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:16:16.100451+0000","last_peered":"2023-10-21T05:16:16.100451+0000","last_clean":"2023-10-21T05:16:16.100451+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:16:16.100451+0000","last_undegraded":"2023-10-21T05:16:16.100451+0000","last_fullsized":"2023-10-21T05:16:16.100451+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":318,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-21T05:14:44.469051+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:14:44.469051+0000","last_peered":"2023-10-21T05:14:44.469051+0000","last_clean":"2023-10-21T05:14:44.469051+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:14:44.469051+0000","last_undegraded":"2023-10-21T05:14:44.469051+0000","last_fullsized":"2023-10-21T05:14:44.469051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29706735635,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2137943,"num_read_kb":8440232,"num_write":2134284,"num_write_kb":8537130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17485811712,"data_stored":37089628198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40206,"ondisk_log_size":40206,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45671,"ondisk_log_size":45671,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739291,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9661412,"kb_used_data":8539416,"kb_used_omap":2489,"kb_used_meta":1119494,"kb_avail":84710428,"statfs":{"total":96636764160,"available":86743478272,"internally_reserved":0,"allocated":8744361984,"data_stored":18545289591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":1146362365},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.96099999999999997,"15min":0.93400000000000005},"min":{"1min":0.32400000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":2.6030000000000002,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":2.589},{"interface":"front","average":{"1min":1.248,"5min":0.91300000000000003,"15min":0.90900000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.2519999999999998,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":2.6259999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.96299999999999997,"15min":1.1120000000000001},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":2.3050000000000002,"5min":4.3879999999999999,"15min":27.196999999999999},"last":2.524},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.0509999999999999,"15min":1.202},"min":{"1min":0.312,"5min":0.249,"15min":0.22800000000000001},"max":{"1min":3.2669999999999999,"5min":9.0020000000000007,"15min":27.242999999999999},"last":2.556}]}]},{"osd":1,"up_from":8,"seq":34359739290,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5879380,"kb_used_data":4192972,"kb_used_omap":1190,"kb_used_meta":1685209,"kb_avail":88492460,"statfs":{"total":96636764160,"available":90616279040,"internally_reserved":0,"allocated":4293603328,"data_stored":9104697675,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":1725654594},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,1,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.71999999999999997,"15min":0.80200000000000005},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":1.804,"5min":5.3920000000000003,"15min":10.711},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70199999999999996,"15min":0.90400000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":2.1030000000000002,"5min":5.4569999999999999,"15min":10.512},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.98599999999999999,"15min":1.0289999999999999},"min":{"1min":0.309,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":4.3540000000000001,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.91900000000000004,"15min":1.069},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.6970000000000001,"5min":5.5030000000000001,"15min":6.1230000000000002},"last":0.88800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6050824,"kb_used_data":4347452,"kb_used_omap":1351,"kb_used_meta":1702008,"kb_avail":88321016,"statfs":{"total":96636764160,"available":90440720384,"internally_reserved":0,"allocated":4451790848,"data_stored":9441231394,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1742856198},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,13,5,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.85899999999999999,"15min":0.876},"min":{"1min":0.27800000000000002,"5min":0.187,"15min":0.183},"max":{"1min":2.8119999999999998,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.091,"5min":0.89800000000000002,"15min":0.93500000000000005},"min":{"1min":0.26500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.956,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.31}]},{"osd":2,"last update":"Sat Oct 21 05:16:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":0.95399999999999996,"15min":0.98699999999999999},"min":{"1min":0.255,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":5.141,"5min":5.141,"15min":5.3179999999999996},"last":2.1840000000000002},{"interface":"front","average":{"1min":1.405,"5min":0.94299999999999995,"15min":0.998},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":5.0270000000000001,"5min":5.0270000000000001,"15min":5.133},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4450254848,"data_stored":9440301056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4291784704,"data_stored":9103646739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8743772160,"data_stored":18545680403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:24.514 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:24.514 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:24.515 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:24.516 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:24.516 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:27.185 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.183+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:16:27.198 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.195+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:16:27.198 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.195+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:16:27.199 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.195+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:16:27.203 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.199+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:16:27.204 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.200+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:16:27.204 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.200+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:16:27.204 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.200+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:16:27.205 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:27.200+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:16:32.775 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.767+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:16:32.783 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.775+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:16:32.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.775+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:16:32.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.775+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:16:32.784 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.781+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:16:32.785 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.781+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:16:32.785 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:16:32.785 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:16:32.781+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:16:44.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:45.073 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:16:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:16:45.094 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2453,"stamp":"2023-10-21T05:16:43.880322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29738993405,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2195272,"num_read_kb":8512622,"num_write":2143152,"num_write_kb":8570383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86090,"ondisk_log_size":86090,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21973632,"kb_used_data":17141668,"kb_used_omap":5031,"kb_used_meta":4826904,"kb_avail":261141888,"statfs":{"total":289910292480,"available":267409293312,"internally_reserved":0,"allocated":17553068032,"data_stored":37154530532,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":4942749765},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,5,13,5,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16011264,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3867,"num_read_kb":15376,"num_write":3892,"num_write_kb":15568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002142"},"pg_stats":[{"pgid":"2.7","version":"143'7157","reported_seq":10565,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:37.204711+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:16:37.204711+0000","last_peered":"2023-10-21T05:16:37.204711+0000","last_clean":"2023-10-21T05:16:37.204711+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:16:37.204711+0000","last_undegraded":"2023-10-21T05:16:37.204711+0000","last_fullsized":"2023-10-21T05:16:37.204711+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7157,"log_dups_size":0,"ondisk_log_size":7157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'3664","reported_seq":7406,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.277341+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:16:36.277341+0000","last_peered":"2023-10-21T05:16:36.277341+0000","last_clean":"2023-10-21T05:16:36.277341+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:16:36.277341+0000","last_undegraded":"2023-10-21T05:16:36.277341+0000","last_fullsized":"2023-10-21T05:16:36.277341+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3664,"log_dups_size":0,"ondisk_log_size":3664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'10678","reported_seq":15652,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:37.205302+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:16:37.205302+0000","last_peered":"2023-10-21T05:16:37.205302+0000","last_clean":"2023-10-21T05:16:37.205302+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:16:37.205302+0000","last_undegraded":"2023-10-21T05:16:37.205302+0000","last_fullsized":"2023-10-21T05:16:37.205302+0000","mapping_epoch":11,"log_start":"47'600","ondisk_log_start":"47'600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'7195","reported_seq":11617,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:37.204664+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:16:37.204664+0000","last_peered":"2023-10-21T05:16:37.204664+0000","last_clean":"2023-10-21T05:16:37.204664+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:16:37.204664+0000","last_undegraded":"2023-10-21T05:16:37.204664+0000","last_fullsized":"2023-10-21T05:16:37.204664+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7195,"log_dups_size":0,"ondisk_log_size":7195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'539776","reported_seq":1066603,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:38.947477+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:16:38.947477+0000","last_peered":"2023-10-21T05:16:38.947477+0000","last_clean":"2023-10-21T05:16:38.947477+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:38.947477+0000","last_undegraded":"2023-10-21T05:16:38.947477+0000","last_fullsized":"2023-10-21T05:16:38.947477+0000","mapping_epoch":14,"log_start":"142'529762","ondisk_log_start":"142'529762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5709,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 621s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7354929152,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":531190,"num_read_kb":2097072,"num_write":530081,"num_write_kb":2120324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'14610","reported_seq":21387,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.097101+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:16:36.097101+0000","last_peered":"2023-10-21T05:16:36.097101+0000","last_clean":"2023-10-21T05:16:36.097101+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:16:36.097101+0000","last_undegraded":"2023-10-21T05:16:36.097101+0000","last_fullsized":"2023-10-21T05:16:36.097101+0000","mapping_epoch":11,"log_start":"80'4600","ondisk_log_start":"80'4600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'545555","reported_seq":1079491,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:39.076043+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:16:39.076043+0000","last_peered":"2023-10-21T05:16:39.076043+0000","last_clean":"2023-10-21T05:16:39.076043+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:16:39.076043+0000","last_undegraded":"2023-10-21T05:16:39.076043+0000","last_fullsized":"2023-10-21T05:16:39.076043+0000","mapping_epoch":14,"log_start":"142'535460","ondisk_log_start":"142'535460","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5166,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 604s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7446130688,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":538381,"num_read_kb":2124904,"num_write":536301,"num_write_kb":2145204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3542,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.105482+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:16:36.105482+0000","last_peered":"2023-10-21T05:16:36.105482+0000","last_clean":"2023-10-21T05:16:36.105482+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:16:36.105482+0000","last_undegraded":"2023-10-21T05:16:36.105482+0000","last_fullsized":"2023-10-21T05:16:36.105482+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'569445","reported_seq":1123864,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:38.935938+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:16:38.935938+0000","last_peered":"2023-10-21T05:16:38.935938+0000","last_clean":"2023-10-21T05:16:38.935938+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:38.935938+0000","last_undegraded":"2023-10-21T05:16:38.935938+0000","last_fullsized":"2023-10-21T05:16:38.935938+0000","mapping_epoch":14,"log_start":"142'559352","ondisk_log_start":"142'559352","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4736,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 618s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7776800768,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":559120,"num_read_kb":2207668,"num_write":560279,"num_write_kb":2241116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7050,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:36.097746+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:16:36.097746+0000","last_peered":"2023-10-21T05:16:36.097746+0000","last_clean":"2023-10-21T05:16:36.097746+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:16:36.097746+0000","last_undegraded":"2023-10-21T05:16:36.097746+0000","last_fullsized":"2023-10-21T05:16:36.097746+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'520202","reported_seq":1032872,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:39.232502+0000","last_change":"2023-10-21T05:16:38.227828+0000","last_active":"2023-10-21T05:16:39.232502+0000","last_peered":"2023-10-21T05:16:39.232502+0000","last_clean":"2023-10-21T05:16:39.232502+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:16:39.232502+0000","last_undegraded":"2023-10-21T05:16:39.232502+0000","last_fullsized":"2023-10-21T05:16:39.232502+0000","mapping_epoch":14,"log_start":"142'510156","ondisk_log_start":"142'510156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7160537107,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":517122,"num_read_kb":2041908,"num_write":515333,"num_write_kb":2061326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'7116","reported_seq":10455,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:37.216943+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:16:37.216943+0000","last_peered":"2023-10-21T05:16:37.216943+0000","last_clean":"2023-10-21T05:16:37.216943+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:16:37.216943+0000","last_undegraded":"2023-10-21T05:16:37.216943+0000","last_fullsized":"2023-10-21T05:16:37.216943+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":319,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:54.480612+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:15:54.480612+0000","last_peered":"2023-10-21T05:15:54.480612+0000","last_clean":"2023-10-21T05:15:54.480612+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:15:54.480612+0000","last_undegraded":"2023-10-21T05:15:54.480612+0000","last_fullsized":"2023-10-21T05:15:54.480612+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29738397715,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2145813,"num_read_kb":8471552,"num_write":2141994,"num_write_kb":8567970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17549029376,"data_stored":37152845862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40248,"ondisk_log_size":40248,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45743,"ondisk_log_size":45743,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739295,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9832476,"kb_used_data":8570320,"kb_used_omap":2489,"kb_used_meta":1259654,"kb_avail":84539364,"statfs":{"total":96636764160,"available":86568308736,"internally_reserved":0,"allocated":8776007680,"data_stored":18576935287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":1289886205},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.96099999999999997,"15min":0.93400000000000005},"min":{"1min":0.32400000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":2.6030000000000002,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":0.441},{"interface":"front","average":{"1min":1.248,"5min":0.91300000000000003,"15min":0.90900000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.2519999999999998,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.96299999999999997,"15min":1.1120000000000001},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":2.3050000000000002,"5min":4.3879999999999999,"15min":27.196999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.0509999999999999,"15min":1.202},"min":{"1min":0.312,"5min":0.249,"15min":0.22800000000000001},"max":{"1min":3.2669999999999999,"5min":9.0020000000000007,"15min":27.242999999999999},"last":0.50700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739294,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5985088,"kb_used_data":4208440,"kb_used_omap":1190,"kb_used_meta":1775449,"kb_avail":88386752,"statfs":{"total":96636764160,"available":90508034048,"internally_reserved":0,"allocated":4309442560,"data_stored":9120536907,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":1818060354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.71999999999999997,"15min":0.80200000000000005},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":1.804,"5min":5.3920000000000003,"15min":10.711},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70199999999999996,"15min":0.90400000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":2.1030000000000002,"5min":5.4569999999999999,"15min":10.512},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.98599999999999999,"15min":1.0289999999999999},"min":{"1min":0.309,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":4.3540000000000001,"5min":4.4969999999999999,"15min":10.167999999999999},"last":0.624},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.91900000000000004,"15min":1.069},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.6970000000000001,"5min":5.5030000000000001,"15min":6.1230000000000002},"last":1.3720000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6156068,"kb_used_data":4362908,"kb_used_omap":1351,"kb_used_meta":1791800,"kb_avail":88215772,"statfs":{"total":96636764160,"available":90332950528,"internally_reserved":0,"allocated":4467617792,"data_stored":9457058338,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1834803206},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,10,4,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.85899999999999999,"15min":0.876},"min":{"1min":0.27800000000000002,"5min":0.187,"15min":0.183},"max":{"1min":2.8119999999999998,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.091,"5min":0.89800000000000002,"15min":0.93500000000000005},"min":{"1min":0.26500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.956,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":1.1419999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:16:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":0.95399999999999996,"15min":0.98699999999999999},"min":{"1min":0.255,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":5.141,"5min":5.141,"15min":5.3179999999999996},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.405,"5min":0.94299999999999995,"15min":0.998},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":5.0270000000000001,"5min":5.0270000000000001,"15min":5.133},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4466135040,"data_stored":9456181248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4307501056,"data_stored":9119363091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8775393280,"data_stored":18577301523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:45.096 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:45.096 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:45.097 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:45.097 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:45.098 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:05.102 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:17:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:17:05.706 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2464,"stamp":"2023-10-21T05:17:03.883868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29770196733,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2203404,"num_read_kb":8545054,"num_write":2150776,"num_write_kb":8600879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86166,"ondisk_log_size":86166,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22308976,"kb_used_data":17202260,"kb_used_omap":5031,"kb_used_meta":5101656,"kb_avail":260806544,"statfs":{"total":289910292480,"available":267065901056,"internally_reserved":0,"allocated":17615114240,"data_stored":37216569017,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":5224095813},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,6,11,4,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15568896,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3978,"num_read_kb":15856,"num_write":3801,"num_write_kb":15204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001674"},"pg_stats":[{"pgid":"2.7","version":"143'7193","reported_seq":10609,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.354984+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:16:57.354984+0000","last_peered":"2023-10-21T05:16:57.354984+0000","last_clean":"2023-10-21T05:16:57.354984+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:16:57.354984+0000","last_undegraded":"2023-10-21T05:16:57.354984+0000","last_fullsized":"2023-10-21T05:16:57.354984+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7193,"log_dups_size":0,"ondisk_log_size":7193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'3682","reported_seq":7429,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.351667+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:16:57.351667+0000","last_peered":"2023-10-21T05:16:57.351667+0000","last_clean":"2023-10-21T05:16:57.351667+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:16:57.351667+0000","last_undegraded":"2023-10-21T05:16:57.351667+0000","last_fullsized":"2023-10-21T05:16:57.351667+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'10732","reported_seq":15718,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.356053+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:16:57.356053+0000","last_peered":"2023-10-21T05:16:57.356053+0000","last_clean":"2023-10-21T05:16:57.356053+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:16:57.356053+0000","last_undegraded":"2023-10-21T05:16:57.356053+0000","last_fullsized":"2023-10-21T05:16:57.356053+0000","mapping_epoch":11,"log_start":"49'700","ondisk_log_start":"49'700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'7231","reported_seq":11661,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.351736+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:16:57.351736+0000","last_peered":"2023-10-21T05:16:57.351736+0000","last_clean":"2023-10-21T05:16:57.351736+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:16:57.351736+0000","last_undegraded":"2023-10-21T05:16:57.351736+0000","last_fullsized":"2023-10-21T05:16:57.351736+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'541820","reported_seq":1070643,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:59.146918+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:16:59.146918+0000","last_peered":"2023-10-21T05:16:59.146918+0000","last_clean":"2023-10-21T05:16:59.146918+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:59.146918+0000","last_undegraded":"2023-10-21T05:16:59.146918+0000","last_fullsized":"2023-10-21T05:16:59.146918+0000","mapping_epoch":14,"log_start":"142'531762","ondisk_log_start":"142'531762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5891,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 641s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7362674688,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":533167,"num_read_kb":2104960,"num_write":531977,"num_write_kb":2127908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'14663","reported_seq":21456,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:56.577058+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:16:56.577058+0000","last_peered":"2023-10-21T05:16:56.577058+0000","last_clean":"2023-10-21T05:16:56.577058+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:16:56.577058+0000","last_undegraded":"2023-10-21T05:16:56.577058+0000","last_fullsized":"2023-10-21T05:16:56.577058+0000","mapping_epoch":11,"log_start":"80'4600","ondisk_log_start":"80'4600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'547598","reported_seq":1083555,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:59.361355+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:16:59.361355+0000","last_peered":"2023-10-21T05:16:59.361355+0000","last_clean":"2023-10-21T05:16:59.361355+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:16:59.361355+0000","last_undegraded":"2023-10-21T05:16:59.361355+0000","last_fullsized":"2023-10-21T05:16:59.361355+0000","mapping_epoch":14,"log_start":"142'537560","ondisk_log_start":"142'537560","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5341,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 624s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7453818880,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":540396,"num_read_kb":2132948,"num_write":538177,"num_write_kb":2152708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3551,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:56.579816+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:16:56.579816+0000","last_peered":"2023-10-21T05:16:56.579816+0000","last_clean":"2023-10-21T05:16:56.579816+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:16:56.579816+0000","last_undegraded":"2023-10-21T05:16:56.579816+0000","last_fullsized":"2023-10-21T05:16:56.579816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'571570","reported_seq":1128121,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:59.136322+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:16:59.136322+0000","last_peered":"2023-10-21T05:16:59.136322+0000","last_clean":"2023-10-21T05:16:59.136322+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:16:59.136322+0000","last_undegraded":"2023-10-21T05:16:59.136322+0000","last_fullsized":"2023-10-21T05:16:59.136322+0000","mapping_epoch":14,"log_start":"142'561552","ondisk_log_start":"142'561552","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4860,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 638s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7785091072,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":561236,"num_read_kb":2216108,"num_write":562300,"num_write_kb":2249200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7067,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:56.579380+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:16:56.579380+0000","last_peered":"2023-10-21T05:16:56.579380+0000","last_clean":"2023-10-21T05:16:56.579380+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:16:56.579380+0000","last_undegraded":"2023-10-21T05:16:56.579380+0000","last_fullsized":"2023-10-21T05:16:56.579380+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'522033","reported_seq":1036737,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:59.358676+0000","last_change":"2023-10-21T05:16:59.185511+0000","last_active":"2023-10-21T05:16:59.358676+0000","last_peered":"2023-10-21T05:16:59.358676+0000","last_clean":"2023-10-21T05:16:59.358676+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:16:59.358676+0000","last_undegraded":"2023-10-21T05:16:59.358676+0000","last_fullsized":"2023-10-21T05:16:59.358676+0000","mapping_epoch":14,"log_start":"142'511956","ondisk_log_start":"142'511956","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7168016403,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":519146,"num_read_kb":2049968,"num_write":517164,"num_write_kb":2068650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'7152","reported_seq":10499,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:57.355757+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:16:57.355757+0000","last_peered":"2023-10-21T05:16:57.355757+0000","last_clean":"2023-10-21T05:16:57.355757+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:16:57.355757+0000","last_undegraded":"2023-10-21T05:16:57.355757+0000","last_fullsized":"2023-10-21T05:16:57.355757+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":319,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:15:54.480612+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:15:54.480612+0000","last_peered":"2023-10-21T05:15:54.480612+0000","last_clean":"2023-10-21T05:15:54.480612+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:15:54.480612+0000","last_undegraded":"2023-10-21T05:15:54.480612+0000","last_fullsized":"2023-10-21T05:15:54.480612+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29769601043,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2153945,"num_read_kb":8503984,"num_write":2149618,"num_write_kb":8598466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17611321344,"data_stored":37215137830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":45876,"ondisk_log_size":45876,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739299,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9984916,"kb_used_data":8600648,"kb_used_omap":2489,"kb_used_meta":1381766,"kb_avail":84386924,"statfs":{"total":96636764160,"available":86412210176,"internally_reserved":0,"allocated":8807063552,"data_stored":18607983436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":1414928893},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.96099999999999997,"15min":0.93400000000000005},"min":{"1min":0.32400000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":2.6030000000000002,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.248,"5min":0.91300000000000003,"15min":0.90900000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.2519999999999998,"5min":4.4219999999999997,"15min":4.4219999999999997},"last":0.36799999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.96299999999999997,"15min":1.1120000000000001},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":2.3050000000000002,"5min":4.3879999999999999,"15min":27.196999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.0509999999999999,"15min":1.202},"min":{"1min":0.312,"5min":0.249,"15min":0.22800000000000001},"max":{"1min":3.2669999999999999,"5min":9.0020000000000007,"15min":27.242999999999999},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739298,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6071984,"kb_used_data":4223144,"kb_used_omap":1190,"kb_used_meta":1847641,"kb_avail":88299856,"statfs":{"total":96636764160,"available":90419052544,"internally_reserved":0,"allocated":4324499456,"data_stored":9135593803,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":1891984962},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,5,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.71999999999999997,"15min":0.80200000000000005},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":1.804,"5min":5.3920000000000003,"15min":10.711},"last":3.742},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70199999999999996,"15min":0.90400000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":2.1030000000000002,"5min":5.4569999999999999,"15min":10.512},"last":3.673}]},{"osd":2,"last update":"Sat Oct 21 05:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.98599999999999999,"15min":1.0289999999999999},"min":{"1min":0.309,"5min":0.19900000000000001,"15min":0.185},"max":{"1min":4.3540000000000001,"5min":4.4969999999999999,"15min":10.167999999999999},"last":3.7080000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.91900000000000004,"15min":1.069},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.6970000000000001,"5min":5.5030000000000001,"15min":6.1230000000000002},"last":0.22}]}]},{"osd":0,"up_from":8,"seq":34359739298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6252076,"kb_used_data":4378468,"kb_used_omap":1351,"kb_used_meta":1872248,"kb_avail":88119764,"statfs":{"total":96636764160,"available":90234638336,"internally_reserved":0,"allocated":4483551232,"data_stored":9472991778,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":1917181958},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,4,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.85899999999999999,"15min":0.876},"min":{"1min":0.27800000000000002,"5min":0.187,"15min":0.183},"max":{"1min":2.8119999999999998,"5min":5.0339999999999998,"15min":7.7140000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.091,"5min":0.89800000000000002,"15min":0.93500000000000005},"min":{"1min":0.26500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.956,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.69299999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:16:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":0.95399999999999996,"15min":0.98699999999999999},"min":{"1min":0.255,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":5.141,"5min":5.141,"15min":5.3179999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.405,"5min":0.94299999999999995,"15min":0.998},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.20100000000000001},"max":{"1min":5.0270000000000001,"5min":5.0270000000000001,"15min":5.133},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4482158592,"data_stored":9472204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4322672640,"data_stored":9134534675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8806490112,"data_stored":18608398355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:05.708 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:05.709 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:05.709 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:05.710 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:05.710 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:15.133 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:17:15.097+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:17:15.133 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:17:15.097+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:17:23.604 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:17:23.603+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:17:25.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:26.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:17:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:17:26.267 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2475,"stamp":"2023-10-21T05:17:25.888495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29810120445,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2213325,"num_read_kb":8584542,"num_write":2160511,"num_write_kb":8639819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86339,"ondisk_log_size":86339,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22785312,"kb_used_data":17279880,"kb_used_omap":5031,"kb_used_meta":5500376,"kb_avail":260330208,"statfs":{"total":289910292480,"available":266578132992,"internally_reserved":0,"allocated":17694597120,"data_stored":37296036451,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":5632385093},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,12,9,6,10,11,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24059904,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5951,"num_read_kb":23716,"num_write":5870,"num_write_kb":23480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002168"},"pg_stats":[{"pgid":"2.7","version":"144'7229","reported_seq":10656,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:24.453345+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:17:24.453345+0000","last_peered":"2023-10-21T05:17:24.453345+0000","last_clean":"2023-10-21T05:17:24.453345+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:17:24.453345+0000","last_undegraded":"2023-10-21T05:17:24.453345+0000","last_fullsized":"2023-10-21T05:17:24.453345+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7229,"log_dups_size":0,"ondisk_log_size":7229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'3700","reported_seq":7453,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:23.254153+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:17:23.254153+0000","last_peered":"2023-10-21T05:17:23.254153+0000","last_clean":"2023-10-21T05:17:23.254153+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:17:23.254153+0000","last_undegraded":"2023-10-21T05:17:23.254153+0000","last_fullsized":"2023-10-21T05:17:23.254153+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'10786","reported_seq":15788,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:24.461525+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:17:24.461525+0000","last_peered":"2023-10-21T05:17:24.461525+0000","last_clean":"2023-10-21T05:17:24.461525+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:17:24.461525+0000","last_undegraded":"2023-10-21T05:17:24.461525+0000","last_fullsized":"2023-10-21T05:17:24.461525+0000","mapping_epoch":11,"log_start":"49'700","ondisk_log_start":"49'700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'7267","reported_seq":11708,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:24.451358+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:17:24.451358+0000","last_peered":"2023-10-21T05:17:24.451358+0000","last_clean":"2023-10-21T05:17:24.451358+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:17:24.451358+0000","last_undegraded":"2023-10-21T05:17:24.451358+0000","last_fullsized":"2023-10-21T05:17:24.451358+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'544441","reported_seq":1075667,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:24.348829+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:17:24.348829+0000","last_peered":"2023-10-21T05:17:24.348829+0000","last_clean":"2023-10-21T05:17:24.348829+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:17:24.348829+0000","last_undegraded":"2023-10-21T05:17:24.348829+0000","last_fullsized":"2023-10-21T05:17:24.348829+0000","mapping_epoch":14,"log_start":"142'534362","ondisk_log_start":"142'534362","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6088,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 666s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7372787712,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":535559,"num_read_kb":2114472,"num_write":534436,"num_write_kb":2137744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'14715","reported_seq":21529,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:23.636805+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:17:23.636805+0000","last_peered":"2023-10-21T05:17:23.636805+0000","last_clean":"2023-10-21T05:17:23.636805+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:17:23.636805+0000","last_undegraded":"2023-10-21T05:17:23.636805+0000","last_fullsized":"2023-10-21T05:17:23.636805+0000","mapping_epoch":11,"log_start":"82'4700","ondisk_log_start":"82'4700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'550107","reported_seq":1088530,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:24.496526+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:17:24.496526+0000","last_peered":"2023-10-21T05:17:24.496526+0000","last_clean":"2023-10-21T05:17:24.496526+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:17:24.496526+0000","last_undegraded":"2023-10-21T05:17:24.496526+0000","last_fullsized":"2023-10-21T05:17:24.496526+0000","mapping_epoch":14,"log_start":"142'540060","ondisk_log_start":"142'540060","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5520,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 649s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7463477248,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":542844,"num_read_kb":2142700,"num_write":540538,"num_write_kb":2162152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3562,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:22.176990+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:17:22.176990+0000","last_peered":"2023-10-21T05:17:22.176990+0000","last_clean":"2023-10-21T05:17:22.176990+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:17:22.176990+0000","last_undegraded":"2023-10-21T05:17:22.176990+0000","last_fullsized":"2023-10-21T05:17:22.176990+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'574245","reported_seq":1133478,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:24.338022+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:17:24.338022+0000","last_peered":"2023-10-21T05:17:24.338022+0000","last_clean":"2023-10-21T05:17:24.338022+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:17:24.338022+0000","last_undegraded":"2023-10-21T05:17:24.338022+0000","last_fullsized":"2023-10-21T05:17:24.338022+0000","mapping_epoch":14,"log_start":"142'564153","ondisk_log_start":"142'564153","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4978,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 663s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7795662848,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":563910,"num_read_kb":2226752,"num_write":564878,"num_write_kb":2259512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7087,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:22.178593+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:17:22.178593+0000","last_peered":"2023-10-21T05:17:22.178593+0000","last_clean":"2023-10-21T05:17:22.178593+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:17:22.178593+0000","last_undegraded":"2023-10-21T05:17:22.178593+0000","last_fullsized":"2023-10-21T05:17:22.178593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'524370","reported_seq":1041492,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:24.559621+0000","last_change":"2023-10-21T05:17:23.630220+0000","last_active":"2023-10-21T05:17:24.559621+0000","last_peered":"2023-10-21T05:17:24.559621+0000","last_clean":"2023-10-21T05:17:24.559621+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:17:24.559621+0000","last_undegraded":"2023-10-21T05:17:24.559621+0000","last_fullsized":"2023-10-21T05:17:24.559621+0000","mapping_epoch":14,"log_start":"142'514356","ondisk_log_start":"142'514356","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7177596947,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":521553,"num_read_kb":2059548,"num_write":519501,"num_write_kb":2077998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'7188","reported_seq":10547,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:24.488244+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:17:24.488244+0000","last_peered":"2023-10-21T05:17:24.488244+0000","last_clean":"2023-10-21T05:17:24.488244+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:17:24.488244+0000","last_undegraded":"2023-10-21T05:17:24.488244+0000","last_fullsized":"2023-10-21T05:17:24.488244+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":320,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:24.004557+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:16:24.004557+0000","last_peered":"2023-10-21T05:16:24.004557+0000","last_clean":"2023-10-21T05:16:24.004557+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:16:24.004557+0000","last_undegraded":"2023-10-21T05:16:24.004557+0000","last_fullsized":"2023-10-21T05:16:24.004557+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29809524755,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2163866,"num_read_kb":8543472,"num_write":2159353,"num_write_kb":8637406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17690910720,"data_stored":37294727206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46008,"ondisk_log_size":46008,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739304,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10255796,"kb_used_data":8639340,"kb_used_omap":2489,"kb_used_meta":1613958,"kb_avail":84116044,"statfs":{"total":96636764160,"available":86134829056,"internally_reserved":0,"allocated":8846684160,"data_stored":18647604044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":1652693501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:16 2023","interfaces":[{"interface":"back","average":{"1min":2.3029999999999999,"5min":1.2110000000000001,"15min":1.026},"min":{"1min":0.191,"5min":0.191,"15min":0.185},"max":{"1min":27.184999999999999,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.496},{"interface":"front","average":{"1min":2.2269999999999999,"5min":1.206,"15min":0.99399999999999999},"min":{"1min":0.29999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":27.251000000000001,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:17:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.86399999999999999,"15min":1.103},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":2.524,"5min":4.2789999999999999,"15min":27.196999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.96299999999999997,"15min":1.2230000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":27.242999999999999},"last":0.745}]}]},{"osd":1,"up_from":8,"seq":34359739303,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6163056,"kb_used_data":4242024,"kb_used_omap":1190,"kb_used_meta":1919833,"kb_avail":88208784,"statfs":{"total":96636764160,"available":90325794816,"internally_reserved":0,"allocated":4343832576,"data_stored":9154919200,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":1965909570},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":3.742,"5min":5.3920000000000003,"15min":10.711},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.90800000000000003},"min":{"1min":0.25700000000000001,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":3.673,"5min":5.4569999999999999,"15min":10.512},"last":0.499}]},{"osd":2,"last update":"Sat Oct 21 05:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.97799999999999998,"15min":1.0109999999999999},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.0339999999999998,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94099999999999995,"15min":1.026},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.544,"5min":5.5030000000000001,"15min":6.1230000000000002},"last":0.376}]}]},{"osd":0,"up_from":8,"seq":34359739303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6366460,"kb_used_data":4398516,"kb_used_omap":1351,"kb_used_meta":1966584,"kb_avail":88005380,"statfs":{"total":96636764160,"available":90117509120,"internally_reserved":0,"allocated":4504080384,"data_stored":9493513207,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":2013782022},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,6,2,5,10,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83699999999999997,"15min":0.86499999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.183},"max":{"1min":4.9870000000000001,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.256,"5min":0.96599999999999997,"15min":0.97499999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":5.5309999999999997,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:17:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.89600000000000002,"15min":0.96199999999999997},"min":{"1min":0.21099999999999999,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5270000000000001,"5min":5.141,"15min":5.3179999999999996},"last":3.5270000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.85799999999999998,"15min":0.97899999999999998},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":2.149,"5min":5.0270000000000001,"15min":5.133},"last":0.27600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4502802432,"data_stored":9492848640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4341923840,"data_stored":9153785875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8846184448,"data_stored":18648092691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:26.269 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:26.269 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:26.269 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:26.270 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:26.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:17:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:17:26.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:17:27.392 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:17:27.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:17:27.965 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:17:27.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:17:28.541 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:17:28.565 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:32.453 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:17:32.448+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:17:33.067 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:17:33.064+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:17:33.067 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:17:33.064+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:17:33.068 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:17:33.064+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:17:48.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:49.094 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:17:49.095 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:17:49.124 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2486,"stamp":"2023-10-21T05:17:47.892326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29841344253,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2221173,"num_read_kb":8615858,"num_write":2168136,"num_write_kb":8670319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86401,"ondisk_log_size":86401,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23122476,"kb_used_data":17340432,"kb_used_omap":5031,"kb_used_meta":5776984,"kb_avail":259993044,"statfs":{"total":289910292480,"available":266232877056,"internally_reserved":0,"allocated":17756602368,"data_stored":37358041699,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":5915631685},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,3,7,11,16,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15925248,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4023,"num_read_kb":16056,"num_write":3891,"num_write_kb":15564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002075"},"pg_stats":[{"pgid":"2.7","version":"144'7259","reported_seq":10694,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:44.486333+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:17:44.486333+0000","last_peered":"2023-10-21T05:17:44.486333+0000","last_clean":"2023-10-21T05:17:44.486333+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:17:44.486333+0000","last_undegraded":"2023-10-21T05:17:44.486333+0000","last_fullsized":"2023-10-21T05:17:44.486333+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'3716","reported_seq":7473,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:42.657605+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:17:42.657605+0000","last_peered":"2023-10-21T05:17:42.657605+0000","last_clean":"2023-10-21T05:17:42.657605+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:17:42.657605+0000","last_undegraded":"2023-10-21T05:17:42.657605+0000","last_fullsized":"2023-10-21T05:17:42.657605+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3716,"log_dups_size":0,"ondisk_log_size":3716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'10832","reported_seq":15847,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:44.506072+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:17:44.506072+0000","last_peered":"2023-10-21T05:17:44.506072+0000","last_clean":"2023-10-21T05:17:44.506072+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:17:44.506072+0000","last_undegraded":"2023-10-21T05:17:44.506072+0000","last_fullsized":"2023-10-21T05:17:44.506072+0000","mapping_epoch":11,"log_start":"49'800","ondisk_log_start":"49'800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'7298","reported_seq":11748,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:44.498394+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:17:44.498394+0000","last_peered":"2023-10-21T05:17:44.498394+0000","last_clean":"2023-10-21T05:17:44.498394+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:17:44.498394+0000","last_undegraded":"2023-10-21T05:17:44.498394+0000","last_fullsized":"2023-10-21T05:17:44.498394+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7298,"log_dups_size":0,"ondisk_log_size":7298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'546449","reported_seq":1079611,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:44.558842+0000","last_change":"2023-10-21T05:06:17.933423+0000","last_active":"2023-10-21T05:17:44.558842+0000","last_peered":"2023-10-21T05:17:44.558842+0000","last_clean":"2023-10-21T05:17:44.558842+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:17:44.558842+0000","last_undegraded":"2023-10-21T05:17:44.558842+0000","last_fullsized":"2023-10-21T05:17:44.558842+0000","mapping_epoch":14,"log_start":"143'536362","ondisk_log_start":"143'536362","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6232,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 686s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7380516864,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":537479,"num_read_kb":2122136,"num_write":536326,"num_write_kb":2145304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'14757","reported_seq":21588,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:42.657152+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:17:42.657152+0000","last_peered":"2023-10-21T05:17:42.657152+0000","last_clean":"2023-10-21T05:17:42.657152+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:17:42.657152+0000","last_undegraded":"2023-10-21T05:17:42.657152+0000","last_fullsized":"2023-10-21T05:17:42.657152+0000","mapping_epoch":11,"log_start":"82'4700","ondisk_log_start":"82'4700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'552134","reported_seq":1092547,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:44.645577+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:17:44.645577+0000","last_peered":"2023-10-21T05:17:44.645577+0000","last_clean":"2023-10-21T05:17:44.645577+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:17:44.645577+0000","last_undegraded":"2023-10-21T05:17:44.645577+0000","last_fullsized":"2023-10-21T05:17:44.645577+0000","mapping_epoch":14,"log_start":"143'542060","ondisk_log_start":"143'542060","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5647,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 669s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7471353856,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":544819,"num_read_kb":2150584,"num_write":542464,"num_write_kb":2169856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3570,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:42.662709+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:17:42.662709+0000","last_peered":"2023-10-21T05:17:42.662709+0000","last_clean":"2023-10-21T05:17:42.662709+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:17:42.662709+0000","last_undegraded":"2023-10-21T05:17:42.662709+0000","last_fullsized":"2023-10-21T05:17:42.662709+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'576355","reported_seq":1137639,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:44.543837+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:17:44.543837+0000","last_peered":"2023-10-21T05:17:44.543837+0000","last_clean":"2023-10-21T05:17:44.543837+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:17:44.543837+0000","last_undegraded":"2023-10-21T05:17:44.543837+0000","last_fullsized":"2023-10-21T05:17:44.543837+0000","mapping_epoch":14,"log_start":"143'566354","ondisk_log_start":"143'566354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5129,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 683s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7803793408,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":565948,"num_read_kb":2234876,"num_write":566864,"num_write_kb":2267456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7103,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:42.667626+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:17:42.667626+0000","last_peered":"2023-10-21T05:17:42.667626+0000","last_clean":"2023-10-21T05:17:42.667626+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:17:42.667626+0000","last_undegraded":"2023-10-21T05:17:42.667626+0000","last_fullsized":"2023-10-21T05:17:42.667626+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'526193","reported_seq":1045241,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:44.668780+0000","last_change":"2023-10-21T05:17:43.535544+0000","last_active":"2023-10-21T05:17:44.668780+0000","last_peered":"2023-10-21T05:17:44.668780+0000","last_clean":"2023-10-21T05:17:44.668780+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:17:44.668780+0000","last_undegraded":"2023-10-21T05:17:44.668780+0000","last_fullsized":"2023-10-21T05:17:44.668780+0000","mapping_epoch":14,"log_start":"142'516156","ondisk_log_start":"142'516156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7185084435,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":523468,"num_read_kb":2067192,"num_write":521324,"num_write_kb":2085290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'7218","reported_seq":10585,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:17:44.500589+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:17:44.500589+0000","last_peered":"2023-10-21T05:17:44.500589+0000","last_clean":"2023-10-21T05:17:44.500589+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:17:44.500589+0000","last_undegraded":"2023-10-21T05:17:44.500589+0000","last_fullsized":"2023-10-21T05:17:44.500589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":320,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:24.004557+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:16:24.004557+0000","last_peered":"2023-10-21T05:16:24.004557+0000","last_clean":"2023-10-21T05:16:24.004557+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:16:24.004557+0000","last_undegraded":"2023-10-21T05:16:24.004557+0000","last_fullsized":"2023-10-21T05:16:24.004557+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29840748563,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2171714,"num_read_kb":8574788,"num_write":2166978,"num_write_kb":8667906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17753264128,"data_stored":37357080614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40199,"ondisk_log_size":40199,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46103,"ondisk_log_size":46103,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739308,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10420536,"kb_used_data":8669484,"kb_used_omap":2489,"kb_used_meta":1748550,"kb_avail":83951304,"statfs":{"total":96636764160,"available":85966135296,"internally_reserved":0,"allocated":8877551616,"data_stored":18678471500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":1790515709},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:16 2023","interfaces":[{"interface":"back","average":{"1min":2.3029999999999999,"5min":1.2110000000000001,"15min":1.026},"min":{"1min":0.191,"5min":0.191,"15min":0.185},"max":{"1min":27.184999999999999,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":1.206,"15min":0.99399999999999999},"min":{"1min":0.29999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":27.251000000000001,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.29699999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:17:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.86399999999999999,"15min":1.103},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":2.524,"5min":4.2789999999999999,"15min":27.196999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.96299999999999997,"15min":1.2230000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":27.242999999999999},"last":0.42099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739307,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6265956,"kb_used_data":4256988,"kb_used_omap":1190,"kb_used_meta":2007769,"kb_avail":88105884,"statfs":{"total":96636764160,"available":90220425216,"internally_reserved":0,"allocated":4359155712,"data_stored":9170242336,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":2055956034},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":3.742,"5min":5.3920000000000003,"15min":10.711},"last":4.1050000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.90800000000000003},"min":{"1min":0.25700000000000001,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":3.673,"5min":5.4569999999999999,"15min":10.512},"last":4.1879999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.97799999999999998,"15min":1.0109999999999999},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.0339999999999998,"5min":5.0339999999999998,"15min":10.167999999999999},"last":4.1459999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94099999999999995,"15min":1.026},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.544,"5min":5.5030000000000001,"15min":6.1230000000000002},"last":0.22}]}]},{"osd":0,"up_from":8,"seq":34359739307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6435984,"kb_used_data":4413960,"kb_used_omap":1351,"kb_used_meta":2020664,"kb_avail":87935856,"statfs":{"total":96636764160,"available":90046316544,"internally_reserved":0,"allocated":4519895040,"data_stored":9509327863,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":2069159942},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,1,5,7,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83699999999999997,"15min":0.86499999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.183},"max":{"1min":4.9870000000000001,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.256,"5min":0.96599999999999997,"15min":0.97499999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":5.5309999999999997,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.42299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:17:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.89600000000000002,"15min":0.96199999999999997},"min":{"1min":0.21099999999999999,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5270000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.85799999999999998,"15min":0.97899999999999998},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":2.149,"5min":5.0270000000000001,"15min":5.133},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4518670336,"data_stored":9508716544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4357283840,"data_stored":9169145875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8877309952,"data_stored":18679218195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:49.126 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:49.127 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:49.127 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:49.127 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:49.128 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:09.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:18:09.693 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:18:09.713 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2497,"stamp":"2023-10-21T05:18:09.498553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29872867069,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2229054,"num_read_kb":8647282,"num_write":2175835,"num_write_kb":8701115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86512,"ondisk_log_size":86512,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23486248,"kb_used_data":17401548,"kb_used_omap":5031,"kb_used_meta":6079640,"kb_avail":259629272,"statfs":{"total":289910292480,"available":265860374528,"internally_reserved":0,"allocated":17819185152,"data_stored":37420624483,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5152699,"internal_metadata":6225551429},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,6,14,20,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15745024,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3889,"num_read_kb":15516,"num_write":3845,"num_write_kb":15380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.604445"},"pg_stats":[{"pgid":"2.7","version":"144'7279","reported_seq":10722,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.546822+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:18:04.546822+0000","last_peered":"2023-10-21T05:18:04.546822+0000","last_clean":"2023-10-21T05:18:04.546822+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:18:04.546822+0000","last_undegraded":"2023-10-21T05:18:04.546822+0000","last_fullsized":"2023-10-21T05:18:04.546822+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'3726","reported_seq":7487,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:03.764533+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:18:03.764533+0000","last_peered":"2023-10-21T05:18:03.764533+0000","last_clean":"2023-10-21T05:18:03.764533+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:18:03.764533+0000","last_undegraded":"2023-10-21T05:18:03.764533+0000","last_fullsized":"2023-10-21T05:18:03.764533+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'10862","reported_seq":15889,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.548668+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:18:04.548668+0000","last_peered":"2023-10-21T05:18:04.548668+0000","last_clean":"2023-10-21T05:18:04.548668+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:18:04.548668+0000","last_undegraded":"2023-10-21T05:18:04.548668+0000","last_fullsized":"2023-10-21T05:18:04.548668+0000","mapping_epoch":11,"log_start":"49'800","ondisk_log_start":"49'800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'7318","reported_seq":11776,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.539394+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:18:04.539394+0000","last_peered":"2023-10-21T05:18:04.539394+0000","last_clean":"2023-10-21T05:18:04.539394+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:18:04.539394+0000","last_undegraded":"2023-10-21T05:18:04.539394+0000","last_fullsized":"2023-10-21T05:18:04.539394+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7318,"log_dups_size":0,"ondisk_log_size":7318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'548309","reported_seq":1083438,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.626302+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:18:04.626302+0000","last_peered":"2023-10-21T05:18:04.626302+0000","last_clean":"2023-10-21T05:18:04.626302+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:18:04.626302+0000","last_undegraded":"2023-10-21T05:18:04.626302+0000","last_fullsized":"2023-10-21T05:18:04.626302+0000","mapping_epoch":14,"log_start":"143'538262","ondisk_log_start":"143'538262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7388004352,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":539438,"num_read_kb":2129952,"num_write":538154,"num_write_kb":2152616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'14813","reported_seq":21660,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:03.138494+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:18:03.138494+0000","last_peered":"2023-10-21T05:18:03.138494+0000","last_clean":"2023-10-21T05:18:03.138494+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:18:03.138494+0000","last_undegraded":"2023-10-21T05:18:03.138494+0000","last_fullsized":"2023-10-21T05:18:03.138494+0000","mapping_epoch":11,"log_start":"82'4800","ondisk_log_start":"82'4800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'554247","reported_seq":1096605,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:04.893087+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:18:04.893087+0000","last_peered":"2023-10-21T05:18:04.893087+0000","last_clean":"2023-10-21T05:18:04.893087+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:18:04.893087+0000","last_undegraded":"2023-10-21T05:18:04.893087+0000","last_fullsized":"2023-10-21T05:18:04.893087+0000","mapping_epoch":14,"log_start":"143'544160","ondisk_log_start":"143'544160","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5808,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 689s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7479447552,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":546748,"num_read_kb":2158276,"num_write":544441,"num_write_kb":2177764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3578,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:03.138368+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:18:03.138368+0000","last_peered":"2023-10-21T05:18:03.138368+0000","last_clean":"2023-10-21T05:18:03.138368+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:18:03.138368+0000","last_undegraded":"2023-10-21T05:18:03.138368+0000","last_fullsized":"2023-10-21T05:18:03.138368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'578485","reported_seq":1141891,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:04.622626+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:18:04.622626+0000","last_peered":"2023-10-21T05:18:04.622626+0000","last_clean":"2023-10-21T05:18:04.622626+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:18:04.622626+0000","last_undegraded":"2023-10-21T05:18:04.622626+0000","last_fullsized":"2023-10-21T05:18:04.622626+0000","mapping_epoch":14,"log_start":"143'568454","ondisk_log_start":"143'568454","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5254,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 703s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7812083712,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":568053,"num_read_kb":2243256,"num_write":568891,"num_write_kb":2275564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7119,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:03.139779+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:18:03.139779+0000","last_peered":"2023-10-21T05:18:03.139779+0000","last_clean":"2023-10-21T05:18:03.139779+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:18:03.139779+0000","last_undegraded":"2023-10-21T05:18:03.139779+0000","last_fullsized":"2023-10-21T05:18:03.139779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'528145","reported_seq":1049089,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:04.890738+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:18:04.890738+0000","last_peered":"2023-10-21T05:18:04.890738+0000","last_clean":"2023-10-21T05:18:04.890738+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:18:04.890738+0000","last_undegraded":"2023-10-21T05:18:04.890738+0000","last_fullsized":"2023-10-21T05:18:04.890738+0000","mapping_epoch":14,"log_start":"143'518056","ondisk_log_start":"143'518056","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":102,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7192735763,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":525356,"num_read_kb":2074728,"num_write":523191,"num_write_kb":2092758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'7238","reported_seq":10613,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-21T05:18:04.562863+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:18:04.562863+0000","last_peered":"2023-10-21T05:18:04.562863+0000","last_clean":"2023-10-21T05:18:04.562863+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:18:04.562863+0000","last_undegraded":"2023-10-21T05:18:04.562863+0000","last_fullsized":"2023-10-21T05:18:04.562863+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":320,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-21T05:16:24.004557+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:16:24.004557+0000","last_peered":"2023-10-21T05:16:24.004557+0000","last_clean":"2023-10-21T05:16:24.004557+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:16:24.004557+0000","last_undegraded":"2023-10-21T05:16:24.004557+0000","last_fullsized":"2023-10-21T05:16:24.004557+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29872271379,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2179595,"num_read_kb":8606212,"num_write":2174677,"num_write_kb":8698702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17816141824,"data_stored":37419958310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40254,"ondisk_log_size":40254,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46159,"ondisk_log_size":46159,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739312,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10591232,"kb_used_data":8699960,"kb_used_omap":2489,"kb_used_meta":1888774,"kb_avail":83780608,"statfs":{"total":96636764160,"available":85791342592,"internally_reserved":0,"allocated":8908759040,"data_stored":18709678924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":1934105085},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:16 2023","interfaces":[{"interface":"back","average":{"1min":2.3029999999999999,"5min":1.2110000000000001,"15min":1.026},"min":{"1min":0.191,"5min":0.191,"15min":0.185},"max":{"1min":27.184999999999999,"5min":27.184999999999999,"15min":27.184999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":2.2269999999999999,"5min":1.206,"15min":0.99399999999999999},"min":{"1min":0.29999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":27.251000000000001,"5min":27.251000000000001,"15min":27.251000000000001},"last":1.153}]},{"osd":1,"last update":"Sat Oct 21 05:17:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.86399999999999999,"15min":1.103},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":2.524,"5min":4.2789999999999999,"15min":27.196999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.96299999999999997,"15min":1.2230000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":27.242999999999999},"last":0.29699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739311,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6353244,"kb_used_data":4272084,"kb_used_omap":1190,"kb_used_meta":2079961,"kb_avail":88018596,"statfs":{"total":96636764160,"available":90131042304,"internally_reserved":0,"allocated":4374614016,"data_stored":9185700640,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":2129880642},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":3.742,"5min":5.3920000000000003,"15min":10.711},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.90800000000000003},"min":{"1min":0.25700000000000001,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":3.673,"5min":5.4569999999999999,"15min":10.512},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:17:09 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.97799999999999998,"15min":1.0109999999999999},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.0339999999999998,"5min":5.0339999999999998,"15min":10.167999999999999},"last":1.006},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94099999999999995,"15min":1.026},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.544,"5min":5.5030000000000001,"15min":6.1230000000000002},"last":1.2230000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6541772,"kb_used_data":4429504,"kb_used_omap":1351,"kb_used_meta":2110904,"kb_avail":87830068,"statfs":{"total":96636764160,"available":89937989632,"internally_reserved":0,"allocated":4535812096,"data_stored":9525244919,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1384442,"internal_metadata":2161565702},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,8,6,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83699999999999997,"15min":0.86499999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.183},"max":{"1min":4.9870000000000001,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.256,"5min":0.96599999999999997,"15min":0.97499999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":5.5309999999999997,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.317}]},{"osd":2,"last update":"Sat Oct 21 05:17:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.89600000000000002,"15min":0.96199999999999997},"min":{"1min":0.21099999999999999,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5270000000000001,"5min":5.141,"15min":5.3179999999999996},"last":1.3939999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.85799999999999998,"15min":0.97899999999999998},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":2.149,"5min":5.0270000000000001,"15min":5.133},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4534456320,"data_stored":9524502528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4373016576,"data_stored":9184878611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8908668928,"data_stored":18710577171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:09.716 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:09.716 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:09.717 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:09.717 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:18.038 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:18.033+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:18:18.039 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:18.033+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:18:21.493 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:21.490+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:18:22.082 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:22.079+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:18:22.083 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:22.079+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:18:25.093 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:18:25.093 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:18:25.094 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:18:25.094 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:18:25.094 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:18:25.094 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:18:25.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:18:25.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:18:25.095 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:18:25.096 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:25.089+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:18:29.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:18:30.234 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:18:30.257 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2508,"stamp":"2023-10-21T05:18:29.899645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29908547325,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2237807,"num_read_kb":8682166,"num_write":2184545,"num_write_kb":8735955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86468,"ondisk_log_size":86468,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22981288,"kb_used_data":17467984,"kb_used_omap":5033,"kb_used_meta":5508246,"kb_avail":260134232,"statfs":{"total":289910292480,"available":266377453568,"internally_reserved":0,"allocated":17887215616,"data_stored":37488631649,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5154379,"internal_metadata":5640444341},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,9,13,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19853312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4888,"num_read_kb":19488,"num_write":4847,"num_write_kb":19388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002040"},"pg_stats":[{"pgid":"2.7","version":"145'7315","reported_seq":10767,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:24.251333+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:18:24.251333+0000","last_peered":"2023-10-21T05:18:24.251333+0000","last_clean":"2023-10-21T05:18:24.251333+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:18:24.251333+0000","last_undegraded":"2023-10-21T05:18:24.251333+0000","last_fullsized":"2023-10-21T05:18:24.251333+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'3745","reported_seq":7511,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:24.235601+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:18:24.235601+0000","last_peered":"2023-10-21T05:18:24.235601+0000","last_clean":"2023-10-21T05:18:24.235601+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:18:24.235601+0000","last_undegraded":"2023-10-21T05:18:24.235601+0000","last_fullsized":"2023-10-21T05:18:24.235601+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3745,"log_dups_size":0,"ondisk_log_size":3745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'10914","reported_seq":15954,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:24.262861+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:18:24.262861+0000","last_peered":"2023-10-21T05:18:24.262861+0000","last_clean":"2023-10-21T05:18:24.262861+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:18:24.262861+0000","last_undegraded":"2023-10-21T05:18:24.262861+0000","last_fullsized":"2023-10-21T05:18:24.262861+0000","mapping_epoch":11,"log_start":"51'900","ondisk_log_start":"51'900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'7355","reported_seq":11822,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:25.087365+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:18:25.087365+0000","last_peered":"2023-10-21T05:18:25.087365+0000","last_clean":"2023-10-21T05:18:25.087365+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:18:25.087365+0000","last_undegraded":"2023-10-21T05:18:25.087365+0000","last_fullsized":"2023-10-21T05:18:25.087365+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"145'550666","reported_seq":1088192,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:29.759880+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:18:29.759880+0000","last_peered":"2023-10-21T05:18:29.759880+0000","last_clean":"2023-10-21T05:18:29.759880+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:18:29.759880+0000","last_undegraded":"2023-10-21T05:18:29.759880+0000","last_fullsized":"2023-10-21T05:18:29.759880+0000","mapping_epoch":14,"log_start":"143'540662","ondisk_log_start":"143'540662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7397666816,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":541833,"num_read_kb":2139492,"num_write":540511,"num_write_kb":2162044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"145'14861","reported_seq":21729,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:28.981481+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:18:28.981481+0000","last_peered":"2023-10-21T05:18:28.981481+0000","last_clean":"2023-10-21T05:18:28.981481+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:18:28.981481+0000","last_undegraded":"2023-10-21T05:18:28.981481+0000","last_fullsized":"2023-10-21T05:18:28.981481+0000","mapping_epoch":11,"log_start":"82'4800","ondisk_log_start":"82'4800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"145'556304","reported_seq":1100598,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:25.054147+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:18:25.054147+0000","last_peered":"2023-10-21T05:18:25.054147+0000","last_clean":"2023-10-21T05:18:25.054147+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:18:25.054147+0000","last_undegraded":"2023-10-21T05:18:25.054147+0000","last_fullsized":"2023-10-21T05:18:25.054147+0000","mapping_epoch":14,"log_start":"143'546261","ondisk_log_start":"143'546261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6009,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 710s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7487176704,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":548670,"num_read_kb":2165936,"num_write":546330,"num_write_kb":2185320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3587,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:23.670259+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:18:23.670259+0000","last_peered":"2023-10-21T05:18:23.670259+0000","last_clean":"2023-10-21T05:18:23.670259+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:18:23.670259+0000","last_undegraded":"2023-10-21T05:18:23.670259+0000","last_fullsized":"2023-10-21T05:18:23.670259+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"145'581168","reported_seq":1147107,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:29.751253+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:18:29.751253+0000","last_peered":"2023-10-21T05:18:29.751253+0000","last_clean":"2023-10-21T05:18:29.751253+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:18:29.751253+0000","last_undegraded":"2023-10-21T05:18:29.751253+0000","last_fullsized":"2023-10-21T05:18:29.751253+0000","mapping_epoch":14,"log_start":"143'571154","ondisk_log_start":"143'571154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5399,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 728s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7822594048,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":570576,"num_read_kb":2253328,"num_write":571457,"num_write_kb":2285828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7137,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:23.631912+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:18:23.631912+0000","last_peered":"2023-10-21T05:18:23.631912+0000","last_clean":"2023-10-21T05:18:23.631912+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:18:23.631912+0000","last_undegraded":"2023-10-21T05:18:23.631912+0000","last_fullsized":"2023-10-21T05:18:23.631912+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"145'530179","reported_seq":1053055,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:25.085041+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:18:25.085041+0000","last_peered":"2023-10-21T05:18:25.085041+0000","last_clean":"2023-10-21T05:18:25.085041+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:18:25.085041+0000","last_undegraded":"2023-10-21T05:18:25.085041+0000","last_fullsized":"2023-10-21T05:18:25.085041+0000","mapping_epoch":14,"log_start":"143'520156","ondisk_log_start":"143'520156","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":267,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7200514067,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":527269,"num_read_kb":2082340,"num_write":525089,"num_write_kb":2100350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'7272","reported_seq":10656,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:24.268958+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:18:24.268958+0000","last_peered":"2023-10-21T05:18:24.268958+0000","last_clean":"2023-10-21T05:18:24.268958+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:18:24.268958+0000","last_undegraded":"2023-10-21T05:18:24.268958+0000","last_fullsized":"2023-10-21T05:18:24.268958+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":322,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:09.504674+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:18:09.504674+0000","last_peered":"2023-10-21T05:18:09.504674+0000","last_clean":"2023-10-21T05:18:09.504674+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:18:09.504674+0000","last_undegraded":"2023-10-21T05:18:09.504674+0000","last_fullsized":"2023-10-21T05:18:09.504674+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29907951635,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2188348,"num_read_kb":8641096,"num_write":2183387,"num_write_kb":8733542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17883172864,"data_stored":37486989350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40084,"ondisk_log_size":40084,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46285,"ondisk_log_size":46285,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739316,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10823396,"kb_used_data":8731740,"kb_used_omap":2489,"kb_used_meta":2089158,"kb_avail":83548444,"statfs":{"total":96636764160,"available":85553606656,"internally_reserved":0,"allocated":8941301760,"data_stored":18742213878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":2139298301},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1679999999999999,"15min":1.03},"min":{"1min":0.186,"5min":0.186,"15min":0.185},"max":{"1min":2.6680000000000001,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.232,"15min":1.02},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.306,"5min":27.251000000000001,"15min":27.251000000000001},"last":1.1399999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:18:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.80500000000000005,"15min":1.0860000000000001},"min":{"1min":0.22700000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":2.3759999999999999,"5min":4.2789999999999999,"15min":27.196999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.87,"5min":0.90500000000000003,"15min":1.1819999999999999},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3370000000000002,"5min":5.1660000000000004,"15min":27.242999999999999},"last":0.92300000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739315,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6404580,"kb_used_data":4287324,"kb_used_omap":1190,"kb_used_meta":2116057,"kb_avail":87967260,"statfs":{"total":96636764160,"available":90078474240,"internally_reserved":0,"allocated":4390219776,"data_stored":9201298634,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":2166842946},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":4.1050000000000004,"5min":5.3920000000000003,"15min":10.711},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.83199999999999996,"15min":0.93600000000000005},"min":{"1min":0.255,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":4.1879999999999997,"5min":5.4569999999999999,"15min":10.512},"last":0.91100000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92900000000000005,"15min":1.012},"min":{"1min":0.28899999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.1459999999999999,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.92300000000000004,"15min":1.024},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":5.9409999999999998,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739316,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5753312,"kb_used_data":4448920,"kb_used_omap":1353,"kb_used_meta":1303030,"kb_avail":88618528,"statfs":{"total":96636764160,"available":90745372672,"internally_reserved":0,"allocated":4555694080,"data_stored":9545119137,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1334303094},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":2.298,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":1.002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.92100000000000004,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":2.335,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:18:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.88300000000000001,"15min":0.95199999999999996},"min":{"1min":0.23100000000000001,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5870000000000002,"5min":5.141,"15min":5.3179999999999996},"last":1.228},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.86499999999999999,"15min":0.96199999999999997},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":4.3620000000000001,"5min":5.0270000000000001,"15min":5.133},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4554620928,"data_stored":9544667136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4388532224,"data_stored":9200394259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8940019712,"data_stored":18741927955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:30.259 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:30.260 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:30.260 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:30.260 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:46.317 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.310+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:18:46.318 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.310+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:18:46.318 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:18:46.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.312+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:18:46.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:18:46.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.315+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:18:46.319 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.315+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:18:46.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.315+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:18:46.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.315+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:18:46.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:46.315+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:18:50.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:50.805 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:18:50.806 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:18:50.826 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2518,"stamp":"2023-10-21T05:18:49.903027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29939955453,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2245543,"num_read_kb":8712990,"num_write":2192216,"num_write_kb":8766639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86608,"ondisk_log_size":86608,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23116620,"kb_used_data":17530096,"kb_used_omap":5033,"kb_used_meta":5581462,"kb_avail":259998900,"statfs":{"total":289910292480,"available":266238873600,"internally_reserved":0,"allocated":17950818304,"data_stored":37552234337,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5154379,"internal_metadata":5715417525},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,5,4,13,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19726336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4638,"num_read_kb":18472,"num_write":4817,"num_write_kb":19268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002038"},"pg_stats":[{"pgid":"2.7","version":"145'7347","reported_seq":10808,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.098841+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:18:44.098841+0000","last_peered":"2023-10-21T05:18:44.098841+0000","last_clean":"2023-10-21T05:18:44.098841+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:18:44.098841+0000","last_undegraded":"2023-10-21T05:18:44.098841+0000","last_fullsized":"2023-10-21T05:18:44.098841+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'3761","reported_seq":7531,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:43.472296+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:18:43.472296+0000","last_peered":"2023-10-21T05:18:43.472296+0000","last_clean":"2023-10-21T05:18:43.472296+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:18:43.472296+0000","last_undegraded":"2023-10-21T05:18:43.472296+0000","last_fullsized":"2023-10-21T05:18:43.472296+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'10965","reported_seq":16017,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.099185+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:18:44.099185+0000","last_peered":"2023-10-21T05:18:44.099185+0000","last_clean":"2023-10-21T05:18:44.099185+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:18:44.099185+0000","last_undegraded":"2023-10-21T05:18:44.099185+0000","last_fullsized":"2023-10-21T05:18:44.099185+0000","mapping_epoch":11,"log_start":"51'900","ondisk_log_start":"51'900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'7387","reported_seq":11862,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.098829+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:18:44.098829+0000","last_peered":"2023-10-21T05:18:44.098829+0000","last_clean":"2023-10-21T05:18:44.098829+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:18:44.098829+0000","last_undegraded":"2023-10-21T05:18:44.098829+0000","last_fullsized":"2023-10-21T05:18:44.098829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7387,"log_dups_size":0,"ondisk_log_size":7387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"145'552574","reported_seq":1091936,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:49.860973+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:18:49.860973+0000","last_peered":"2023-10-21T05:18:49.860973+0000","last_clean":"2023-10-21T05:18:49.860973+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:18:49.860973+0000","last_undegraded":"2023-10-21T05:18:49.860973+0000","last_fullsized":"2023-10-21T05:18:49.860973+0000","mapping_epoch":14,"log_start":"144'542562","ondisk_log_start":"144'542562","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7405481984,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":543669,"num_read_kb":2146812,"num_write":542419,"num_write_kb":2169676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"145'14905","reported_seq":21789,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:49.590100+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:18:49.590100+0000","last_peered":"2023-10-21T05:18:49.590100+0000","last_clean":"2023-10-21T05:18:49.590100+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:18:49.590100+0000","last_undegraded":"2023-10-21T05:18:49.590100+0000","last_fullsized":"2023-10-21T05:18:49.590100+0000","mapping_epoch":11,"log_start":"87'4900","ondisk_log_start":"87'4900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"145'558386","reported_seq":1104700,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:45.199868+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:18:45.199868+0000","last_peered":"2023-10-21T05:18:45.199868+0000","last_clean":"2023-10-21T05:18:45.199868+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:18:45.199868+0000","last_undegraded":"2023-10-21T05:18:45.199868+0000","last_fullsized":"2023-10-21T05:18:45.199868+0000","mapping_epoch":14,"log_start":"144'548361","ondisk_log_start":"144'548361","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6152,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 730s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7495221248,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":550681,"num_read_kb":2173952,"num_write":548297,"num_write_kb":2193188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3595,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.124106+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:18:44.124106+0000","last_peered":"2023-10-21T05:18:44.124106+0000","last_clean":"2023-10-21T05:18:44.124106+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:18:44.124106+0000","last_undegraded":"2023-10-21T05:18:44.124106+0000","last_fullsized":"2023-10-21T05:18:44.124106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"145'583210","reported_seq":1151229,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:49.863236+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:18:49.863236+0000","last_peered":"2023-10-21T05:18:49.863236+0000","last_clean":"2023-10-21T05:18:49.863236+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:18:49.863236+0000","last_undegraded":"2023-10-21T05:18:49.863236+0000","last_fullsized":"2023-10-21T05:18:49.863236+0000","mapping_epoch":14,"log_start":"144'573154","ondisk_log_start":"144'573154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5542,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 748s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7830491136,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":572644,"num_read_kb":2261576,"num_write":573383,"num_write_kb":2293532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7153,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.101533+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:18:44.101533+0000","last_peered":"2023-10-21T05:18:44.101533+0000","last_clean":"2023-10-21T05:18:44.101533+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:18:44.101533+0000","last_undegraded":"2023-10-21T05:18:44.101533+0000","last_fullsized":"2023-10-21T05:18:44.101533+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"145'532179","reported_seq":1056887,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:45.184680+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:18:45.184680+0000","last_peered":"2023-10-21T05:18:45.184680+0000","last_clean":"2023-10-21T05:18:45.184680+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:18:45.184680+0000","last_undegraded":"2023-10-21T05:18:45.184680+0000","last_fullsized":"2023-10-21T05:18:45.184680+0000","mapping_epoch":14,"log_start":"144'522157","ondisk_log_start":"144'522157","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":430,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 53s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7208165395,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":529090,"num_read_kb":2089580,"num_write":526959,"num_write_kb":2107830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'7306","reported_seq":10698,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:44.099201+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:18:44.099201+0000","last_peered":"2023-10-21T05:18:44.099201+0000","last_clean":"2023-10-21T05:18:44.099201+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:18:44.099201+0000","last_undegraded":"2023-10-21T05:18:44.099201+0000","last_fullsized":"2023-10-21T05:18:44.099201+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":322,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:09.504674+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:18:09.504674+0000","last_peered":"2023-10-21T05:18:09.504674+0000","last_clean":"2023-10-21T05:18:09.504674+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:18:09.504674+0000","last_undegraded":"2023-10-21T05:18:09.504674+0000","last_fullsized":"2023-10-21T05:18:09.504674+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29939359763,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2196084,"num_read_kb":8671920,"num_write":2191058,"num_write_kb":8764226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17946431488,"data_stored":37550247974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40115,"ondisk_log_size":40115,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46394,"ondisk_log_size":46394,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739320,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10993236,"kb_used_data":8762888,"kb_used_omap":2489,"kb_used_meta":2227846,"kb_avail":83378604,"statfs":{"total":96636764160,"available":85379690496,"internally_reserved":0,"allocated":8973197312,"data_stored":18774109430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":2281314813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1679999999999999,"15min":1.03},"min":{"1min":0.186,"5min":0.186,"15min":0.185},"max":{"1min":2.6680000000000001,"5min":27.184999999999999,"15min":27.184999999999999},"last":2.3420000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.232,"15min":1.02},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.306,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:18:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.80500000000000005,"15min":1.0860000000000001},"min":{"1min":0.22700000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":2.3759999999999999,"5min":4.2789999999999999,"15min":27.196999999999999},"last":2.2989999999999999},{"interface":"front","average":{"1min":0.87,"5min":0.90500000000000003,"15min":1.1819999999999999},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3370000000000002,"5min":5.1660000000000004,"15min":27.242999999999999},"last":2.2559999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739319,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6337636,"kb_used_data":4302748,"kb_used_omap":1190,"kb_used_meta":2033689,"kb_avail":88034204,"statfs":{"total":96636764160,"available":90147024896,"internally_reserved":0,"allocated":4406013952,"data_stored":9217092810,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":2082498114},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,0,1,3,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":4.1050000000000004,"5min":5.3920000000000003,"15min":10.711},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.83199999999999996,"15min":0.93600000000000005},"min":{"1min":0.255,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":4.1879999999999997,"5min":5.4569999999999999,"15min":10.512},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92900000000000005,"15min":1.012},"min":{"1min":0.28899999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.1459999999999999,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.307},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.92300000000000004,"15min":1.024},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":5.9409999999999998,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.32400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739320,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5785748,"kb_used_data":4464460,"kb_used_omap":1353,"kb_used_meta":1319926,"kb_avail":88586092,"statfs":{"total":96636764160,"available":90712158208,"internally_reserved":0,"allocated":4571607040,"data_stored":9561032097,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1351604598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":2.298,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.92100000000000004,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":2.335,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:18:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.88300000000000001,"15min":0.95199999999999996},"min":{"1min":0.23100000000000001,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5870000000000002,"5min":5.141,"15min":5.3179999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.86499999999999999,"15min":0.96199999999999997},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":4.3620000000000001,"5min":5.0270000000000001,"15min":5.133},"last":0.64100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4570341376,"data_stored":9560387584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4404248576,"data_stored":9216110611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8971841536,"data_stored":18773749779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:50.828 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:50.829 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:50.829 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:50.830 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:54.490 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:54.488+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:18:54.491 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:54.488+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:18:54.491 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:54.488+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:18:54.491 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:18:54.488+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:18:56.712 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.711+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:18:56.716 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.714+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:18:56.720 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.718+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:18:56.721 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.719+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:18:56.726 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.724+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:18:56.726 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.724+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:18:56.726 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.724+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:18:56.727 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:18:56.724+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:19:10.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:11.365 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:19:11.367 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:19:11.389 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2528,"stamp":"2023-10-21T05:19:09.907038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29966989053,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2252226,"num_read_kb":8739634,"num_write":2198810,"num_write_kb":8793015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86829,"ondisk_log_size":86829,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23345300,"kb_used_data":17585788,"kb_used_omap":5033,"kb_used_meta":5754454,"kb_avail":259770220,"statfs":{"total":289910292480,"available":266004705280,"internally_reserved":0,"allocated":18007846912,"data_stored":37609262945,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5154379,"internal_metadata":5892561333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,10,3,7,9,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15532032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3933,"num_read_kb":15688,"num_write":3792,"num_write_kb":15168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002711"},"pg_stats":[{"pgid":"2.7","version":"145'7369","reported_seq":10838,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.962113+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:19:04.962113+0000","last_peered":"2023-10-21T05:19:04.962113+0000","last_clean":"2023-10-21T05:19:04.962113+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:19:04.962113+0000","last_undegraded":"2023-10-21T05:19:04.962113+0000","last_fullsized":"2023-10-21T05:19:04.962113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'3772","reported_seq":7547,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.952918+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:19:04.952918+0000","last_peered":"2023-10-21T05:19:04.952918+0000","last_clean":"2023-10-21T05:19:04.952918+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:19:04.952918+0000","last_undegraded":"2023-10-21T05:19:04.952918+0000","last_fullsized":"2023-10-21T05:19:04.952918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'10998","reported_seq":16062,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.967622+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:19:04.967622+0000","last_peered":"2023-10-21T05:19:04.967622+0000","last_clean":"2023-10-21T05:19:04.967622+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:19:04.967622+0000","last_undegraded":"2023-10-21T05:19:04.967622+0000","last_fullsized":"2023-10-21T05:19:04.967622+0000","mapping_epoch":11,"log_start":"51'900","ondisk_log_start":"51'900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'7409","reported_seq":11892,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.960884+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:19:04.960884+0000","last_peered":"2023-10-21T05:19:04.960884+0000","last_clean":"2023-10-21T05:19:04.960884+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:19:04.960884+0000","last_undegraded":"2023-10-21T05:19:04.960884+0000","last_fullsized":"2023-10-21T05:19:04.960884+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"145'554043","reported_seq":1094772,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.972896+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:19:04.972896+0000","last_peered":"2023-10-21T05:19:04.972896+0000","last_clean":"2023-10-21T05:19:04.972896+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:19:04.972896+0000","last_undegraded":"2023-10-21T05:19:04.972896+0000","last_fullsized":"2023-10-21T05:19:04.972896+0000","mapping_epoch":14,"log_start":"144'543962","ondisk_log_start":"144'543962","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7411499008,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":545039,"num_read_kb":2152268,"num_write":543888,"num_write_kb":2175552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"145'14950","reported_seq":21846,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.577179+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:19:04.577179+0000","last_peered":"2023-10-21T05:19:04.577179+0000","last_clean":"2023-10-21T05:19:04.577179+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:19:04.577179+0000","last_undegraded":"2023-10-21T05:19:04.577179+0000","last_fullsized":"2023-10-21T05:19:04.577179+0000","mapping_epoch":11,"log_start":"87'4900","ondisk_log_start":"87'4900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"145'560443","reported_seq":1108716,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:05.463948+0000","last_change":"2023-10-21T05:06:34.984656+0000","last_active":"2023-10-21T05:19:05.463948+0000","last_peered":"2023-10-21T05:19:05.463948+0000","last_clean":"2023-10-21T05:19:05.463948+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:19:05.463948+0000","last_undegraded":"2023-10-21T05:19:05.463948+0000","last_fullsized":"2023-10-21T05:19:05.463948+0000","mapping_epoch":14,"log_start":"144'550361","ondisk_log_start":"144'550361","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6312,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 750s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7503110144,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":552623,"num_read_kb":2181684,"num_write":550223,"num_write_kb":2200892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3603,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.582254+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:19:04.582254+0000","last_peered":"2023-10-21T05:19:04.582254+0000","last_clean":"2023-10-21T05:19:04.582254+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:19:04.582254+0000","last_undegraded":"2023-10-21T05:19:04.582254+0000","last_fullsized":"2023-10-21T05:19:04.582254+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"145'584760","reported_seq":1154296,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:04.951332+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:19:04.951332+0000","last_peered":"2023-10-21T05:19:04.951332+0000","last_clean":"2023-10-21T05:19:04.951332+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:19:04.951332+0000","last_undegraded":"2023-10-21T05:19:04.951332+0000","last_fullsized":"2023-10-21T05:19:04.951332+0000","mapping_epoch":14,"log_start":"144'574754","ondisk_log_start":"144'574754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5645,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 764s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7836499968,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":574155,"num_read_kb":2267612,"num_write":574849,"num_write_kb":2299396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7169,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.577651+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:19:04.577651+0000","last_peered":"2023-10-21T05:19:04.577651+0000","last_clean":"2023-10-21T05:19:04.577651+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:19:04.577651+0000","last_undegraded":"2023-10-21T05:19:04.577651+0000","last_fullsized":"2023-10-21T05:19:04.577651+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"145'534070","reported_seq":1060654,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:05.482765+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:19:05.482765+0000","last_peered":"2023-10-21T05:19:05.482765+0000","last_clean":"2023-10-21T05:19:05.482765+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:19:05.482765+0000","last_undegraded":"2023-10-21T05:19:05.482765+0000","last_fullsized":"2023-10-21T05:19:05.482765+0000","mapping_epoch":14,"log_start":"144'524058","ondisk_log_start":"144'524058","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":624,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 74s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7215284243,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":530950,"num_read_kb":2097000,"num_write":528692,"num_write_kb":2114762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'7328","reported_seq":10728,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:19:04.983012+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:19:04.983012+0000","last_peered":"2023-10-21T05:19:04.983012+0000","last_clean":"2023-10-21T05:19:04.983012+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:19:04.983012+0000","last_undegraded":"2023-10-21T05:19:04.983012+0000","last_fullsized":"2023-10-21T05:19:04.983012+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7328,"log_dups_size":0,"ondisk_log_size":7328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":322,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:09.504674+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:18:09.504674+0000","last_peered":"2023-10-21T05:18:09.504674+0000","last_clean":"2023-10-21T05:18:09.504674+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:18:09.504674+0000","last_undegraded":"2023-10-21T05:18:09.504674+0000","last_fullsized":"2023-10-21T05:18:09.504674+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29966393363,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2202767,"num_read_kb":8698564,"num_write":2197652,"num_write_kb":8790602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18004303872,"data_stored":37608120358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46549,"ondisk_log_size":46549,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739324,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11127152,"kb_used_data":8792680,"kb_used_omap":2489,"kb_used_meta":2331974,"kb_avail":83244688,"statfs":{"total":96636764160,"available":85242560512,"internally_reserved":0,"allocated":9003704320,"data_stored":18804616438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2549251,"internal_metadata":2387941885},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1679999999999999,"15min":1.03},"min":{"1min":0.186,"5min":0.186,"15min":0.185},"max":{"1min":2.6680000000000001,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.232,"15min":1.02},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.306,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.376}]},{"osd":1,"last update":"Sat Oct 21 05:18:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.80500000000000005,"15min":1.0860000000000001},"min":{"1min":0.22700000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":2.3759999999999999,"5min":4.2789999999999999,"15min":27.196999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.87,"5min":0.90500000000000003,"15min":1.1819999999999999},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3370000000000002,"5min":5.1660000000000004,"15min":27.242999999999999},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739323,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6442060,"kb_used_data":4316932,"kb_used_omap":1190,"kb_used_meta":2123929,"kb_avail":87929780,"statfs":{"total":96636764160,"available":90040094720,"internally_reserved":0,"allocated":4420538368,"data_stored":9231617226,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":2174903874},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,7,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":4.1050000000000004,"5min":5.3920000000000003,"15min":10.711},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.83199999999999996,"15min":0.93600000000000005},"min":{"1min":0.255,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":4.1879999999999997,"5min":5.4569999999999999,"15min":10.512},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92900000000000005,"15min":1.012},"min":{"1min":0.28899999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.1459999999999999,"5min":5.0339999999999998,"15min":10.167999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.92300000000000004,"15min":1.024},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":5.9409999999999998,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":1.079}]}]},{"osd":0,"up_from":8,"seq":34359739323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5776088,"kb_used_data":4476176,"kb_used_omap":1353,"kb_used_meta":1298550,"kb_avail":88595752,"statfs":{"total":96636764160,"available":90722050048,"internally_reserved":0,"allocated":4583604224,"data_stored":9573029281,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1329715574},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":2.298,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.92100000000000004,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":2.335,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:18:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.88300000000000001,"15min":0.95199999999999996},"min":{"1min":0.23100000000000001,"5min":0.20100000000000001,"15min":0.16900000000000001},"max":{"1min":3.5870000000000002,"5min":5.141,"15min":5.3179999999999996},"last":3.8799999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.86499999999999999,"15min":0.96199999999999997},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":4.3620000000000001,"5min":5.0270000000000001,"15min":5.133},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4582354944,"data_stored":9572401152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4419235840,"data_stored":9231097875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9002713088,"data_stored":18804621331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:11.392 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:11.392 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:11.392 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:11.393 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:14.391 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:19:14.391 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.388+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:19:14.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.388+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:19:14.394 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.390+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:19:14.394 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.390+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:19:14.396 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.394+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:19:14.399 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.397+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:19:14.399 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.397+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:19:14.400 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.397+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:19:14.400 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:14.397+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:19:31.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:31.630 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:19:31.630 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:19:31.631 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:19:31.631 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:19:31.631 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:19:31.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:19:31.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:19:31.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:19:31.632 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:31.627+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:19:31.930 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:19:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:19:31.949 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2540,"stamp":"2023-10-21T05:19:31.910703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30007113469,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2261892,"num_read_kb":8778166,"num_write":2208606,"num_write_kb":8832199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86907,"ondisk_log_size":86907,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23232664,"kb_used_data":17663552,"kb_used_omap":5234,"kb_used_meta":5563853,"kb_avail":259882856,"statfs":{"total":289910292480,"available":266120044544,"internally_reserved":0,"allocated":18087477248,"data_stored":37688869854,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5359911,"internal_metadata":5697386201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,6,10,14,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23781376,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5996,"num_read_kb":23900,"num_write":5812,"num_write_kb":23248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002078"},"pg_stats":[{"pgid":"2.7","version":"146'7403","reported_seq":10883,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.178336+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:19:30.178336+0000","last_peered":"2023-10-21T05:19:30.178336+0000","last_clean":"2023-10-21T05:19:30.178336+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:19:30.178336+0000","last_undegraded":"2023-10-21T05:19:30.178336+0000","last_fullsized":"2023-10-21T05:19:30.178336+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'3789","reported_seq":7570,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:27.326911+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:19:27.326911+0000","last_peered":"2023-10-21T05:19:27.326911+0000","last_clean":"2023-10-21T05:19:27.326911+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:19:27.326911+0000","last_undegraded":"2023-10-21T05:19:27.326911+0000","last_fullsized":"2023-10-21T05:19:27.326911+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'11049","reported_seq":16129,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.178112+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:19:30.178112+0000","last_peered":"2023-10-21T05:19:30.178112+0000","last_clean":"2023-10-21T05:19:30.178112+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:19:30.178112+0000","last_undegraded":"2023-10-21T05:19:30.178112+0000","last_fullsized":"2023-10-21T05:19:30.178112+0000","mapping_epoch":11,"log_start":"51'1000","ondisk_log_start":"51'1000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'7443","reported_seq":11937,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.178923+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:19:30.178923+0000","last_peered":"2023-10-21T05:19:30.178923+0000","last_clean":"2023-10-21T05:19:30.178923+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:19:30.178923+0000","last_undegraded":"2023-10-21T05:19:30.178923+0000","last_fullsized":"2023-10-21T05:19:30.178923+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7443,"log_dups_size":0,"ondisk_log_size":7443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'556432","reported_seq":1099599,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.042537+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:19:30.042537+0000","last_peered":"2023-10-21T05:19:30.042537+0000","last_clean":"2023-10-21T05:19:30.042537+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:19:30.042537+0000","last_undegraded":"2023-10-21T05:19:30.042537+0000","last_fullsized":"2023-10-21T05:19:30.042537+0000","mapping_epoch":14,"log_start":"144'546362","ondisk_log_start":"144'546362","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7421280256,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":547472,"num_read_kb":2161972,"num_write":546277,"num_write_kb":2185108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'15011","reported_seq":21924,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:27.306594+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:19:27.306594+0000","last_peered":"2023-10-21T05:19:27.306594+0000","last_clean":"2023-10-21T05:19:27.306594+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:19:27.306594+0000","last_undegraded":"2023-10-21T05:19:27.306594+0000","last_fullsized":"2023-10-21T05:19:27.306594+0000","mapping_epoch":11,"log_start":"89'5000","ondisk_log_start":"89'5000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'562918","reported_seq":1113537,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.631373+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:19:30.631373+0000","last_peered":"2023-10-21T05:19:30.631373+0000","last_clean":"2023-10-21T05:19:30.631373+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:19:30.631373+0000","last_undegraded":"2023-10-21T05:19:30.631373+0000","last_fullsized":"2023-10-21T05:19:30.631373+0000","mapping_epoch":14,"log_start":"144'552861","ondisk_log_start":"144'552861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7513149440,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":554962,"num_read_kb":2191000,"num_write":552677,"num_write_kb":2210708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3615,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.191832+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:19:30.191832+0000","last_peered":"2023-10-21T05:19:30.191832+0000","last_clean":"2023-10-21T05:19:30.191832+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:19:30.191832+0000","last_undegraded":"2023-10-21T05:19:30.191832+0000","last_fullsized":"2023-10-21T05:19:30.191832+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'587444","reported_seq":1159591,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:30.058275+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:19:30.058275+0000","last_peered":"2023-10-21T05:19:30.058275+0000","last_clean":"2023-10-21T05:19:30.058275+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:19:30.058275+0000","last_undegraded":"2023-10-21T05:19:30.058275+0000","last_fullsized":"2023-10-21T05:19:30.058275+0000","mapping_epoch":14,"log_start":"144'577354","ondisk_log_start":"144'577354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5792,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 789s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7847014400,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":576746,"num_read_kb":2277952,"num_write":577412,"num_write_kb":2309648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7190,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.181803+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:19:30.181803+0000","last_peered":"2023-10-21T05:19:30.181803+0000","last_clean":"2023-10-21T05:19:30.181803+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:19:30.181803+0000","last_undegraded":"2023-10-21T05:19:30.181803+0000","last_fullsized":"2023-10-21T05:19:30.181803+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'536669","reported_seq":1065575,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:30.642937+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:19:30.642937+0000","last_peered":"2023-10-21T05:19:30.642937+0000","last_clean":"2023-10-21T05:19:30.642937+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:19:30.642937+0000","last_undegraded":"2023-10-21T05:19:30.642937+0000","last_fullsized":"2023-10-21T05:19:30.642937+0000","mapping_epoch":14,"log_start":"144'526658","ondisk_log_start":"144'526658","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":879,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 99s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7225073683,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":533253,"num_read_kb":2106172,"num_write":531082,"num_write_kb":2124322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'7362","reported_seq":10774,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:30.178175+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:19:30.178175+0000","last_peered":"2023-10-21T05:19:30.178175+0000","last_clean":"2023-10-21T05:19:30.178175+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:19:30.178175+0000","last_undegraded":"2023-10-21T05:19:30.178175+0000","last_fullsized":"2023-10-21T05:19:30.178175+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":323,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:25.066102+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:18:25.066102+0000","last_peered":"2023-10-21T05:18:25.066102+0000","last_clean":"2023-10-21T05:18:25.066102+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:18:25.066102+0000","last_undegraded":"2023-10-21T05:18:25.066102+0000","last_fullsized":"2023-10-21T05:18:25.066102+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30006517779,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2212433,"num_read_kb":8737096,"num_write":2207448,"num_write_kb":8829786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18084470784,"data_stored":37688287270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40228,"ondisk_log_size":40228,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46580,"ondisk_log_size":46580,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739329,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10894420,"kb_used_data":8831372,"kb_used_omap":2690,"kb_used_meta":2060349,"kb_avail":83477420,"statfs":{"total":96636764160,"available":85480878080,"internally_reserved":0,"allocated":9043324928,"data_stored":18844229237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2754783,"internal_metadata":2109798177},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2709999999999999,"15min":1.0640000000000001},"min":{"1min":0.27400000000000002,"5min":0.186,"15min":0.185},"max":{"1min":4.5659999999999998,"5min":27.184999999999999,"15min":27.184999999999999},"last":2.7599999999999998},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.3240000000000001,"15min":1.03},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.0620000000000003,"5min":27.251000000000001,"15min":27.251000000000001},"last":2.6890000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":0.91700000000000004,"15min":1.1100000000000001},"min":{"1min":0.245,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7690000000000001,"5min":4.7690000000000001,"15min":27.196999999999999},"last":2.7330000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.0349999999999999,"15min":1.2030000000000001},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.0129999999999999,"5min":5.1660000000000004,"15min":27.242999999999999},"last":2.7919999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739328,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6521096,"kb_used_data":4336256,"kb_used_omap":1190,"kb_used_meta":2183641,"kb_avail":87850744,"statfs":{"total":96636764160,"available":89959161856,"internally_reserved":0,"allocated":4440326144,"data_stored":9251397193,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1219006,"internal_metadata":2236048962},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,11,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72099999999999997,"15min":0.80600000000000005},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":2.3479999999999999,"5min":4.1050000000000004,"15min":10.711},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.81399999999999995,"15min":0.89800000000000002},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":2.431,"5min":4.1879999999999997,"15min":10.512},"last":0.59999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.89900000000000002,"15min":0.98999999999999999},"min":{"1min":0.28399999999999997,"5min":0.245,"15min":0.19900000000000001},"max":{"1min":2.3900000000000001,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.873,"15min":0.997},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.137,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.376}]}]},{"osd":0,"up_from":8,"seq":34359739328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5817148,"kb_used_data":4495924,"kb_used_omap":1353,"kb_used_meta":1319862,"kb_avail":88554692,"statfs":{"total":96636764160,"available":90680004608,"internally_reserved":0,"allocated":4603826176,"data_stored":9593243424,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1351539062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.77400000000000002,"15min":0.83499999999999996},"min":{"1min":0.189,"5min":0.187,"15min":0.187},"max":{"1min":1.052,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.85999999999999999,"15min":0.91700000000000004},"min":{"1min":0.26700000000000002,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":0.73199999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:24 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.97899999999999998,"15min":0.94999999999999996},"min":{"1min":0.21299999999999999,"5min":0.20100000000000001,"15min":0.19600000000000001},"max":{"1min":4.6280000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.874,"5min":0.89100000000000001,"15min":0.94099999999999995},"min":{"1min":0.28399999999999997,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":2.0579999999999998,"5min":5.0270000000000001,"15min":5.133},"last":4.702}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4602630144,"data_stored":9592676352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4439080960,"data_stored":9250942995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9042759680,"data_stored":18844667923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:31.951 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:31.952 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:31.952 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:40.416 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:40.413+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:19:40.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:40.413+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:19:40.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:40.413+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:19:40.417 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:40.413+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:19:40.418 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:40.413+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:19:48.858 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.855+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:19:48.859 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.855+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:19:48.871 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:19:48.871 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:19:48.872 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:19:48.872 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:19:48.872 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:19:48.873 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:19:48.873 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:19:48.873 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:48.863+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:19:51.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:52.489 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:19:52.490 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:19:52.510 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2550,"stamp":"2023-10-21T05:19:51.914231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30039066365,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2269642,"num_read_kb":8809070,"num_write":2216402,"num_write_kb":8863383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87079,"ondisk_log_size":87079,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22552964,"kb_used_data":17726252,"kb_used_omap":5235,"kb_used_meta":4821452,"kb_avail":260562556,"statfs":{"total":289910292480,"available":266816057344,"internally_reserved":0,"allocated":18151682048,"data_stored":37753074654,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5361590,"internal_metadata":4937166922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,5,9,11,10,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24043520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5756,"num_read_kb":22960,"num_write":5868,"num_write_kb":23472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002063"},"pg_stats":[{"pgid":"2.7","version":"146'7434","reported_seq":10922,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.660323+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:19:50.660323+0000","last_peered":"2023-10-21T05:19:50.660323+0000","last_clean":"2023-10-21T05:19:50.660323+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:19:50.660323+0000","last_undegraded":"2023-10-21T05:19:50.660323+0000","last_fullsized":"2023-10-21T05:19:50.660323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7434,"log_dups_size":0,"ondisk_log_size":7434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'3805","reported_seq":7590,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:48.860465+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:19:48.860465+0000","last_peered":"2023-10-21T05:19:48.860465+0000","last_clean":"2023-10-21T05:19:48.860465+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:19:48.860465+0000","last_undegraded":"2023-10-21T05:19:48.860465+0000","last_fullsized":"2023-10-21T05:19:48.860465+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'11095","reported_seq":16188,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.660249+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:19:50.660249+0000","last_peered":"2023-10-21T05:19:50.660249+0000","last_clean":"2023-10-21T05:19:50.660249+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:19:50.660249+0000","last_undegraded":"2023-10-21T05:19:50.660249+0000","last_fullsized":"2023-10-21T05:19:50.660249+0000","mapping_epoch":11,"log_start":"51'1000","ondisk_log_start":"51'1000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'7474","reported_seq":11977,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.660173+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:19:50.660173+0000","last_peered":"2023-10-21T05:19:50.660173+0000","last_clean":"2023-10-21T05:19:50.660173+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:19:50.660173+0000","last_undegraded":"2023-10-21T05:19:50.660173+0000","last_fullsized":"2023-10-21T05:19:50.660173+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7474,"log_dups_size":0,"ondisk_log_size":7474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'558299","reported_seq":1103421,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.107261+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:19:50.107261+0000","last_peered":"2023-10-21T05:19:50.107261+0000","last_clean":"2023-10-21T05:19:50.107261+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:19:50.107261+0000","last_undegraded":"2023-10-21T05:19:50.107261+0000","last_fullsized":"2023-10-21T05:19:50.107261+0000","mapping_epoch":14,"log_start":"144'548262","ondisk_log_start":"144'548262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7428931584,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":549426,"num_read_kb":2169760,"num_write":548144,"num_write_kb":2192576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'15068","reported_seq":21998,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:49.731045+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:19:49.731045+0000","last_peered":"2023-10-21T05:19:49.731045+0000","last_clean":"2023-10-21T05:19:49.731045+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:19:49.731045+0000","last_undegraded":"2023-10-21T05:19:49.731045+0000","last_fullsized":"2023-10-21T05:19:49.731045+0000","mapping_epoch":11,"log_start":"89'5000","ondisk_log_start":"89'5000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'564830","reported_seq":1117358,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.747984+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:19:50.747984+0000","last_peered":"2023-10-21T05:19:50.747984+0000","last_clean":"2023-10-21T05:19:50.747984+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:19:50.747984+0000","last_undegraded":"2023-10-21T05:19:50.747984+0000","last_fullsized":"2023-10-21T05:19:50.747984+0000","mapping_epoch":14,"log_start":"145'554761","ondisk_log_start":"145'554761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7520989184,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":556871,"num_read_kb":2198616,"num_write":554589,"num_write_kb":2218356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3623,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.658794+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:19:50.658794+0000","last_peered":"2023-10-21T05:19:50.658794+0000","last_clean":"2023-10-21T05:19:50.658794+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:19:50.658794+0000","last_undegraded":"2023-10-21T05:19:50.658794+0000","last_fullsized":"2023-10-21T05:19:50.658794+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'589662","reported_seq":1163810,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:50.097947+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:19:50.097947+0000","last_peered":"2023-10-21T05:19:50.097947+0000","last_clean":"2023-10-21T05:19:50.097947+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:19:50.097947+0000","last_undegraded":"2023-10-21T05:19:50.097947+0000","last_fullsized":"2023-10-21T05:19:50.097947+0000","mapping_epoch":14,"log_start":"145'579654","ondisk_log_start":"145'579654","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5928,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 809s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7855669248,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":578736,"num_read_kb":2285892,"num_write":579521,"num_write_kb":2318084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7206,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.657317+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:19:50.657317+0000","last_peered":"2023-10-21T05:19:50.657317+0000","last_clean":"2023-10-21T05:19:50.657317+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:19:50.657317+0000","last_undegraded":"2023-10-21T05:19:50.657317+0000","last_fullsized":"2023-10-21T05:19:50.657317+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'538732","reported_seq":1069546,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:50.768660+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:19:50.768660+0000","last_peered":"2023-10-21T05:19:50.768660+0000","last_clean":"2023-10-21T05:19:50.768660+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:19:50.768660+0000","last_undegraded":"2023-10-21T05:19:50.768660+0000","last_fullsized":"2023-10-21T05:19:50.768660+0000","mapping_epoch":14,"log_start":"145'528658","ondisk_log_start":"145'528658","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1069,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 119s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7232880659,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":535150,"num_read_kb":2113732,"num_write":532990,"num_write_kb":2131954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'7393","reported_seq":10813,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.660488+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:19:50.660488+0000","last_peered":"2023-10-21T05:19:50.660488+0000","last_clean":"2023-10-21T05:19:50.660488+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:19:50.660488+0000","last_undegraded":"2023-10-21T05:19:50.660488+0000","last_fullsized":"2023-10-21T05:19:50.660488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":323,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:25.066102+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:18:25.066102+0000","last_peered":"2023-10-21T05:18:25.066102+0000","last_clean":"2023-10-21T05:18:25.066102+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:18:25.066102+0000","last_undegraded":"2023-10-21T05:18:25.066102+0000","last_fullsized":"2023-10-21T05:18:25.066102+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30038470675,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2220183,"num_read_kb":8768000,"num_write":2215244,"num_write_kb":8860970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18148265984,"data_stored":37752082470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46792,"ondisk_log_size":46792,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739333,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10959128,"kb_used_data":8862736,"kb_used_omap":2690,"kb_used_meta":2093693,"kb_avail":83412712,"statfs":{"total":96636764160,"available":85414617088,"internally_reserved":0,"allocated":9075441664,"data_stored":18876345973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2754783,"internal_metadata":2143942433},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2709999999999999,"15min":1.0640000000000001},"min":{"1min":0.27400000000000002,"5min":0.186,"15min":0.185},"max":{"1min":4.5659999999999998,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.37},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.3240000000000001,"15min":1.03},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.0620000000000003,"5min":27.251000000000001,"15min":27.251000000000001},"last":3.863}]},{"osd":1,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":0.91700000000000004,"15min":1.1100000000000001},"min":{"1min":0.245,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7690000000000001,"5min":4.7690000000000001,"15min":27.196999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.0349999999999999,"15min":1.2030000000000001},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.0129999999999999,"5min":5.1660000000000004,"15min":27.242999999999999},"last":0.96999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739332,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5670380,"kb_used_data":4351524,"kb_used_omap":1192,"kb_used_meta":1317655,"kb_avail":88701460,"statfs":{"total":96636764160,"available":90830295040,"internally_reserved":0,"allocated":4455960576,"data_stored":9267031625,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1349279667},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,6,5,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72099999999999997,"15min":0.80600000000000005},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":2.3479999999999999,"5min":4.1050000000000004,"15min":10.711},"last":1.5920000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.81399999999999995,"15min":0.89800000000000002},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":2.431,"5min":4.1879999999999997,"15min":10.512},"last":1.639}]},{"osd":2,"last update":"Sat Oct 21 05:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.89900000000000002,"15min":0.98999999999999999},"min":{"1min":0.28399999999999997,"5min":0.245,"15min":0.19900000000000001},"max":{"1min":2.3900000000000001,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.873,"15min":0.997},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.137,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.37}]}]},{"osd":0,"up_from":8,"seq":34359739332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5923456,"kb_used_data":4511992,"kb_used_omap":1353,"kb_used_meta":1410102,"kb_avail":88448384,"statfs":{"total":96636764160,"available":90571145216,"internally_reserved":0,"allocated":4620279808,"data_stored":9609697056,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1443944822},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.77400000000000002,"15min":0.83499999999999996},"min":{"1min":0.189,"5min":0.187,"15min":0.187},"max":{"1min":1.052,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.85999999999999999,"15min":0.91700000000000004},"min":{"1min":0.26700000000000002,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":0.73199999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":1.03}]},{"osd":2,"last update":"Sat Oct 21 05:19:24 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.97899999999999998,"15min":0.94999999999999996},"min":{"1min":0.21299999999999999,"5min":0.20100000000000001,"15min":0.19600000000000001},"max":{"1min":4.6280000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.89100000000000001,"15min":0.94099999999999995},"min":{"1min":0.28399999999999997,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":2.0579999999999998,"5min":5.0270000000000001,"15min":5.133},"last":1.5389999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4618919936,"data_stored":9608966144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4454719488,"data_stored":9266581523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9074626560,"data_stored":18876534803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:52.512 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:52.512 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:52.513 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:53.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.185+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:19:53.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.185+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:19:53.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.185+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:19:53.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.185+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:19:53.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.185+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:19:53.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.191+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:19:53.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.191+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:19:53.195 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.191+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:19:53.196 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.191+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:19:53.196 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:19:53.191+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:20:12.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:13.045 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:20:13.046 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:20:13.061 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2560,"stamp":"2023-10-21T05:20:11.917689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30072063741,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2277823,"num_read_kb":8841694,"num_write":2224458,"num_write_kb":8895607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87108,"ondisk_log_size":87108,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21555784,"kb_used_data":17791212,"kb_used_omap":5237,"kb_used_meta":3759306,"kb_avail":261559736,"statfs":{"total":289910292480,"available":267837169664,"internally_reserved":0,"allocated":18218201088,"data_stored":37819593694,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":3849529786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,4,10,16,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24883200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6138,"num_read_kb":24472,"num_write":6074,"num_write_kb":24296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002061"},"pg_stats":[{"pgid":"2.7","version":"146'7464","reported_seq":10958,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:09.505494+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:20:09.505494+0000","last_peered":"2023-10-21T05:20:09.505494+0000","last_clean":"2023-10-21T05:20:09.505494+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:20:09.505494+0000","last_undegraded":"2023-10-21T05:20:09.505494+0000","last_fullsized":"2023-10-21T05:20:09.505494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'3820","reported_seq":7608,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:07.699058+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:20:07.699058+0000","last_peered":"2023-10-21T05:20:07.699058+0000","last_clean":"2023-10-21T05:20:07.699058+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:20:07.699058+0000","last_undegraded":"2023-10-21T05:20:07.699058+0000","last_fullsized":"2023-10-21T05:20:07.699058+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'11140","reported_seq":16242,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:09.521464+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:20:09.521464+0000","last_peered":"2023-10-21T05:20:09.521464+0000","last_clean":"2023-10-21T05:20:09.521464+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:20:09.521464+0000","last_undegraded":"2023-10-21T05:20:09.521464+0000","last_fullsized":"2023-10-21T05:20:09.521464+0000","mapping_epoch":11,"log_start":"53'1100","ondisk_log_start":"53'1100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'7504","reported_seq":12013,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:09.506608+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:20:09.506608+0000","last_peered":"2023-10-21T05:20:09.506608+0000","last_clean":"2023-10-21T05:20:09.506608+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:20:09.506608+0000","last_undegraded":"2023-10-21T05:20:09.506608+0000","last_fullsized":"2023-10-21T05:20:09.506608+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'560302","reported_seq":1107450,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:10.226680+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:20:10.226680+0000","last_peered":"2023-10-21T05:20:10.226680+0000","last_clean":"2023-10-21T05:20:10.226680+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:20:10.226680+0000","last_undegraded":"2023-10-21T05:20:10.226680+0000","last_fullsized":"2023-10-21T05:20:10.226680+0000","mapping_epoch":14,"log_start":"145'550262","ondisk_log_start":"145'550262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7437123584,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":551452,"num_read_kb":2177844,"num_write":550147,"num_write_kb":2200588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'15124","reported_seq":22070,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:10.054080+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:20:10.054080+0000","last_peered":"2023-10-21T05:20:10.054080+0000","last_clean":"2023-10-21T05:20:10.054080+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:20:10.054080+0000","last_undegraded":"2023-10-21T05:20:10.054080+0000","last_fullsized":"2023-10-21T05:20:10.054080+0000","mapping_epoch":11,"log_start":"91'5100","ondisk_log_start":"91'5100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'566846","reported_seq":1121377,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:11.034794+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:20:11.034794+0000","last_peered":"2023-10-21T05:20:11.034794+0000","last_clean":"2023-10-21T05:20:11.034794+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:20:11.034794+0000","last_undegraded":"2023-10-21T05:20:11.034794+0000","last_fullsized":"2023-10-21T05:20:11.034794+0000","mapping_epoch":14,"log_start":"145'556761","ondisk_log_start":"145'556761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7529246720,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":558874,"num_read_kb":2206612,"num_write":556605,"num_write_kb":2226420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3629,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:06.025340+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:20:06.025340+0000","last_peered":"2023-10-21T05:20:06.025340+0000","last_clean":"2023-10-21T05:20:06.025340+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:20:06.025340+0000","last_undegraded":"2023-10-21T05:20:06.025340+0000","last_fullsized":"2023-10-21T05:20:06.025340+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'591816","reported_seq":1168120,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:10.228012+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:20:10.228012+0000","last_peered":"2023-10-21T05:20:10.228012+0000","last_clean":"2023-10-21T05:20:10.228012+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:20:10.228012+0000","last_undegraded":"2023-10-21T05:20:10.228012+0000","last_fullsized":"2023-10-21T05:20:10.228012+0000","mapping_epoch":14,"log_start":"145'581754","ondisk_log_start":"145'581754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6063,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 829s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7864053760,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":580882,"num_read_kb":2294448,"num_write":581567,"num_write_kb":2326268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7219,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:06.018488+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:20:06.018488+0000","last_peered":"2023-10-21T05:20:06.018488+0000","last_clean":"2023-10-21T05:20:06.018488+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:20:06.018488+0000","last_undegraded":"2023-10-21T05:20:06.018488+0000","last_fullsized":"2023-10-21T05:20:06.018488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'540882","reported_seq":1073707,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:11.055835+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:20:11.055835+0000","last_peered":"2023-10-21T05:20:11.055835+0000","last_clean":"2023-10-21T05:20:11.055835+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:20:11.055835+0000","last_undegraded":"2023-10-21T05:20:11.055835+0000","last_fullsized":"2023-10-21T05:20:11.055835+0000","mapping_epoch":14,"log_start":"145'530858","ondisk_log_start":"145'530858","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1267,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 139s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7241043987,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":537156,"num_read_kb":2121720,"num_write":534981,"num_write_kb":2139918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'7423","reported_seq":10849,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:09.515306+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:20:09.515306+0000","last_peered":"2023-10-21T05:20:09.515306+0000","last_clean":"2023-10-21T05:20:09.515306+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:20:09.515306+0000","last_undegraded":"2023-10-21T05:20:09.515306+0000","last_fullsized":"2023-10-21T05:20:09.515306+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":323,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-21T05:18:25.066102+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:18:25.066102+0000","last_peered":"2023-10-21T05:18:25.066102+0000","last_clean":"2023-10-21T05:18:25.066102+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:18:25.066102+0000","last_undegraded":"2023-10-21T05:18:25.066102+0000","last_fullsized":"2023-10-21T05:18:25.066102+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30071468051,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2228364,"num_read_kb":8800624,"num_write":2223300,"num_write_kb":8893194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18214076416,"data_stored":37817892902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40211,"ondisk_log_size":40211,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46798,"ondisk_log_size":46798,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739337,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9794936,"kb_used_data":8894896,"kb_used_omap":2691,"kb_used_meta":897340,"kb_avail":84576904,"statfs":{"total":96636764160,"available":86606749696,"internally_reserved":0,"allocated":9108373504,"data_stored":18909277813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":918876305},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2709999999999999,"15min":1.0640000000000001},"min":{"1min":0.27400000000000002,"5min":0.186,"15min":0.185},"max":{"1min":4.5659999999999998,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.3240000000000001,"15min":1.03},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.0620000000000003,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.20399999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":0.91700000000000004,"15min":1.1100000000000001},"min":{"1min":0.245,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.7690000000000001,"5min":4.7690000000000001,"15min":27.196999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.0349999999999999,"15min":1.2030000000000001},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.0129999999999999,"5min":5.1660000000000004,"15min":27.242999999999999},"last":1.7090000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739336,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5767128,"kb_used_data":4368208,"kb_used_omap":1192,"kb_used_meta":1397719,"kb_avail":88604712,"statfs":{"total":96636764160,"available":90731225088,"internally_reserved":0,"allocated":4473044992,"data_stored":9284116041,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1431265203},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72099999999999997,"15min":0.80600000000000005},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.16400000000000001},"max":{"1min":2.3479999999999999,"5min":4.1050000000000004,"15min":10.711},"last":1.423},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.81399999999999995,"15min":0.89800000000000002},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":2.431,"5min":4.1879999999999997,"15min":10.512},"last":1.4830000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.89900000000000002,"15min":0.98999999999999999},"min":{"1min":0.28399999999999997,"5min":0.245,"15min":0.19900000000000001},"max":{"1min":2.3900000000000001,"5min":5.0339999999999998,"15min":10.167999999999999},"last":1.5880000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.873,"15min":0.997},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.137,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":1.9330000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5993720,"kb_used_data":4528108,"kb_used_omap":1353,"kb_used_meta":1464246,"kb_avail":88378120,"statfs":{"total":96636764160,"available":90499194880,"internally_reserved":0,"allocated":4636782592,"data_stored":9626199840,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1499388278},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,8,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.77400000000000002,"15min":0.83499999999999996},"min":{"1min":0.189,"5min":0.187,"15min":0.187},"max":{"1min":1.052,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.85999999999999999,"15min":0.91700000000000004},"min":{"1min":0.26700000000000002,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":0.73199999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:19:24 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.97899999999999998,"15min":0.94999999999999996},"min":{"1min":0.21299999999999999,"5min":0.20100000000000001,"15min":0.19600000000000001},"max":{"1min":4.6280000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.89100000000000001,"15min":0.94099999999999995},"min":{"1min":0.28399999999999997,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":2.0579999999999998,"5min":5.0270000000000001,"15min":5.133},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4635504640,"data_stored":9625550848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4471136256,"data_stored":9282998291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9107435520,"data_stored":18909343763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:13.063 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:13.064 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:13.064 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:33.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:33.602 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:20:33.603 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:20:33.623 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2570,"stamp":"2023-10-21T05:20:31.921500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30105048829,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2285718,"num_read_kb":8873162,"num_write":2232510,"num_write_kb":8927815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87242,"ondisk_log_size":87242,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21922856,"kb_used_data":17855696,"kb_used_omap":5237,"kb_used_meta":4061898,"kb_avail":261192664,"statfs":{"total":289910292480,"available":267461287936,"internally_reserved":0,"allocated":18284232704,"data_stored":37885625310,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":4159383994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,2,12,12,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24780800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5970,"num_read_kb":23788,"num_write":6047,"num_write_kb":24188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002112"},"pg_stats":[{"pgid":"2.7","version":"146'7490","reported_seq":10993,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:30.999125+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:20:30.999125+0000","last_peered":"2023-10-21T05:20:30.999125+0000","last_clean":"2023-10-21T05:20:30.999125+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:20:30.999125+0000","last_undegraded":"2023-10-21T05:20:30.999125+0000","last_fullsized":"2023-10-21T05:20:30.999125+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'3833","reported_seq":7625,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:28.965331+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:20:28.965331+0000","last_peered":"2023-10-21T05:20:28.965331+0000","last_clean":"2023-10-21T05:20:28.965331+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:20:28.965331+0000","last_undegraded":"2023-10-21T05:20:28.965331+0000","last_fullsized":"2023-10-21T05:20:28.965331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'11179","reported_seq":16293,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:31.005884+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:20:31.005884+0000","last_peered":"2023-10-21T05:20:31.005884+0000","last_clean":"2023-10-21T05:20:31.005884+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:20:31.005884+0000","last_undegraded":"2023-10-21T05:20:31.005884+0000","last_fullsized":"2023-10-21T05:20:31.005884+0000","mapping_epoch":11,"log_start":"53'1100","ondisk_log_start":"53'1100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'7530","reported_seq":12047,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:30.998819+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:20:30.998819+0000","last_peered":"2023-10-21T05:20:30.998819+0000","last_clean":"2023-10-21T05:20:30.998819+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:20:30.998819+0000","last_undegraded":"2023-10-21T05:20:30.998819+0000","last_fullsized":"2023-10-21T05:20:30.998819+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'562307","reported_seq":1111403,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:30.341653+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:20:30.341653+0000","last_peered":"2023-10-21T05:20:30.341653+0000","last_clean":"2023-10-21T05:20:30.341653+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:20:30.341653+0000","last_undegraded":"2023-10-21T05:20:30.341653+0000","last_fullsized":"2023-10-21T05:20:30.341653+0000","mapping_epoch":14,"log_start":"145'552262","ondisk_log_start":"145'552262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7445340160,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":553402,"num_read_kb":2185608,"num_write":552152,"num_write_kb":2208608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'15173","reported_seq":22135,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:27.904364+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:20:27.904364+0000","last_peered":"2023-10-21T05:20:27.904364+0000","last_clean":"2023-10-21T05:20:27.904364+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:20:27.904364+0000","last_undegraded":"2023-10-21T05:20:27.904364+0000","last_fullsized":"2023-10-21T05:20:27.904364+0000","mapping_epoch":11,"log_start":"91'5100","ondisk_log_start":"91'5100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'568868","reported_seq":1125411,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:31.121500+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:20:31.121500+0000","last_peered":"2023-10-21T05:20:31.121500+0000","last_clean":"2023-10-21T05:20:31.121500+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:20:31.121500+0000","last_undegraded":"2023-10-21T05:20:31.121500+0000","last_fullsized":"2023-10-21T05:20:31.121500+0000","mapping_epoch":14,"log_start":"145'558861","ondisk_log_start":"145'558861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7537524736,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":560886,"num_read_kb":2214648,"num_write":558627,"num_write_kb":2234508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3637,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.500235+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:20:26.500235+0000","last_peered":"2023-10-21T05:20:26.500235+0000","last_clean":"2023-10-21T05:20:26.500235+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:20:26.500235+0000","last_undegraded":"2023-10-21T05:20:26.500235+0000","last_fullsized":"2023-10-21T05:20:26.500235+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'593994","reported_seq":1172394,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:30.341681+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:20:30.341681+0000","last_peered":"2023-10-21T05:20:30.341681+0000","last_clean":"2023-10-21T05:20:30.341681+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:20:30.341681+0000","last_undegraded":"2023-10-21T05:20:30.341681+0000","last_fullsized":"2023-10-21T05:20:30.341681+0000","mapping_epoch":14,"log_start":"145'583954","ondisk_log_start":"145'583954","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6172,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 849s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7872606208,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":582967,"num_read_kb":2302756,"num_write":583655,"num_write_kb":2334620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7235,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.499744+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:20:26.499744+0000","last_peered":"2023-10-21T05:20:26.499744+0000","last_clean":"2023-10-21T05:20:26.499744+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:20:26.499744+0000","last_undegraded":"2023-10-21T05:20:26.499744+0000","last_fullsized":"2023-10-21T05:20:26.499744+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'542932","reported_seq":1077627,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:31.114362+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:20:31.114362+0000","last_peered":"2023-10-21T05:20:31.114362+0000","last_clean":"2023-10-21T05:20:31.114362+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:20:31.114362+0000","last_undegraded":"2023-10-21T05:20:31.114362+0000","last_fullsized":"2023-10-21T05:20:31.114362+0000","mapping_epoch":14,"log_start":"145'532858","ondisk_log_start":"145'532858","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1406,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 159s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7248982035,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":539004,"num_read_kb":2129080,"num_write":536918,"num_write_kb":2147666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'7449","reported_seq":10883,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:31.017410+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:20:31.017410+0000","last_peered":"2023-10-21T05:20:31.017410+0000","last_clean":"2023-10-21T05:20:31.017410+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:20:31.017410+0000","last_undegraded":"2023-10-21T05:20:31.017410+0000","last_fullsized":"2023-10-21T05:20:31.017410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":324,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:19:19.513035+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:19:19.513035+0000","last_peered":"2023-10-21T05:19:19.513035+0000","last_clean":"2023-10-21T05:19:19.513035+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:19:19.513035+0000","last_undegraded":"2023-10-21T05:19:19.513035+0000","last_fullsized":"2023-10-21T05:19:19.513035+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30104453139,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2236259,"num_read_kb":8832092,"num_write":2231352,"num_write_kb":8925402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18279948288,"data_stored":37883764774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40166,"ondisk_log_size":40166,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46977,"ondisk_log_size":46977,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739341,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9967476,"kb_used_data":8927212,"kb_used_omap":2691,"kb_used_meta":1037564,"kb_avail":84404364,"statfs":{"total":96636764160,"available":86430068736,"internally_reserved":0,"allocated":9141465088,"data_stored":18942369397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1062465681},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":1.236,"15min":1.0529999999999999},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.0950000000000002,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.253},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.284,"15min":1.0129999999999999},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.863,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.29199999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86699999999999999,"15min":0.97699999999999998},"min":{"1min":0.222,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":3.254,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.216},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0589999999999999,"15min":1.1080000000000001},"min":{"1min":0.26500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.2229999999999999,"5min":5.1660000000000004,"15min":10.253},"last":0.33200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739340,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5855060,"kb_used_data":4384012,"kb_used_omap":1192,"kb_used_meta":1469847,"kb_avail":88516780,"statfs":{"total":96636764160,"available":90641182720,"internally_reserved":0,"allocated":4489228288,"data_stored":9300299337,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1505124275},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.752,"15min":0.81799999999999995},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":3.3919999999999999,"5min":4.1050000000000004,"15min":10.711},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":0.88900000000000001},"min":{"1min":0.24199999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":3.3519999999999999,"5min":4.1879999999999997,"15min":10.512},"last":0.497}]},{"osd":2,"last update":"Sat Oct 21 05:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89500000000000002,"15min":0.96699999999999997},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.19900000000000001},"max":{"1min":3.4279999999999999,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.84399999999999997,"15min":0.95999999999999996},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.9330000000000001,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6100320,"kb_used_data":4544472,"kb_used_omap":1353,"kb_used_meta":1554486,"kb_avail":88271520,"statfs":{"total":96636764160,"available":90390036480,"internally_reserved":0,"allocated":4653539328,"data_stored":9642956576,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1591794038},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,9,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.81299999999999994,"15min":0.82499999999999996},"min":{"1min":0.23200000000000001,"5min":0.189,"15min":0.187},"max":{"1min":4.5570000000000004,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.016,"5min":0.89300000000000002,"15min":0.90600000000000003},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":5.968,"5min":5.968,"15min":6.4050000000000002},"last":0.443}]},{"osd":2,"last update":"Sat Oct 21 05:20:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.93100000000000005,"15min":0.93999999999999995},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":2.3580000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.31},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.008,"15min":0.98799999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":4.702,"5min":5.0270000000000001,"15min":5.133},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4652265472,"data_stored":9642311680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4487348224,"data_stored":9299210259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9140334592,"data_stored":18942242835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:33.624 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:33.625 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:33.625 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:33.848 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:20:33.845+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:20:33.848 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:20:33.845+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:20:53.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:54.159 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:20:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:20:54.185 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2582,"stamp":"2023-10-21T05:20:53.925296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30136927997,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2293298,"num_read_kb":8903354,"num_write":2240285,"num_write_kb":8958915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87287,"ondisk_log_size":87287,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22180472,"kb_used_data":17917472,"kb_used_omap":5237,"kb_used_meta":4257738,"kb_avail":260935048,"statfs":{"total":289910292480,"available":267197489152,"internally_reserved":0,"allocated":18347491328,"data_stored":37948860378,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":4359924154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,3,13,7,13,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16068608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3753,"num_read_kb":14936,"num_write":3913,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002084"},"pg_stats":[{"pgid":"2.7","version":"147'7512","reported_seq":11024,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:50.493777+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:20:50.493777+0000","last_peered":"2023-10-21T05:20:50.493777+0000","last_clean":"2023-10-21T05:20:50.493777+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:20:50.493777+0000","last_undegraded":"2023-10-21T05:20:50.493777+0000","last_fullsized":"2023-10-21T05:20:50.493777+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'3844","reported_seq":7641,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:49.596341+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:20:49.596341+0000","last_peered":"2023-10-21T05:20:49.596341+0000","last_clean":"2023-10-21T05:20:49.596341+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:20:49.596341+0000","last_undegraded":"2023-10-21T05:20:49.596341+0000","last_fullsized":"2023-10-21T05:20:49.596341+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3844,"log_dups_size":0,"ondisk_log_size":3844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'11211","reported_seq":16338,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:50.506868+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:20:50.506868+0000","last_peered":"2023-10-21T05:20:50.506868+0000","last_clean":"2023-10-21T05:20:50.506868+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:20:50.506868+0000","last_undegraded":"2023-10-21T05:20:50.506868+0000","last_fullsized":"2023-10-21T05:20:50.506868+0000","mapping_epoch":11,"log_start":"53'1200","ondisk_log_start":"53'1200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'7552","reported_seq":12078,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:50.500584+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:20:50.500584+0000","last_peered":"2023-10-21T05:20:50.500584+0000","last_clean":"2023-10-21T05:20:50.500584+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:20:50.500584+0000","last_undegraded":"2023-10-21T05:20:50.500584+0000","last_fullsized":"2023-10-21T05:20:50.500584+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"147'564216","reported_seq":1115209,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:50.485319+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:20:50.485319+0000","last_peered":"2023-10-21T05:20:50.485319+0000","last_clean":"2023-10-21T05:20:50.485319+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:20:50.485319+0000","last_undegraded":"2023-10-21T05:20:50.485319+0000","last_fullsized":"2023-10-21T05:20:50.485319+0000","mapping_epoch":14,"log_start":"145'554162","ondisk_log_start":"145'554162","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7453155328,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":555297,"num_read_kb":2193152,"num_write":554061,"num_write_kb":2216244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'15225","reported_seq":22204,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:50.405388+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:20:50.405388+0000","last_peered":"2023-10-21T05:20:50.405388+0000","last_clean":"2023-10-21T05:20:50.405388+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:20:50.405388+0000","last_undegraded":"2023-10-21T05:20:50.405388+0000","last_fullsized":"2023-10-21T05:20:50.405388+0000","mapping_epoch":11,"log_start":"91'5200","ondisk_log_start":"91'5200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'570728","reported_seq":1129220,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:51.182743+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:20:51.182743+0000","last_peered":"2023-10-21T05:20:51.182743+0000","last_clean":"2023-10-21T05:20:51.182743+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:20:51.182743+0000","last_undegraded":"2023-10-21T05:20:51.182743+0000","last_fullsized":"2023-10-21T05:20:51.182743+0000","mapping_epoch":14,"log_start":"145'560661","ondisk_log_start":"145'560661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7545151488,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":562835,"num_read_kb":2222408,"num_write":560487,"num_write_kb":2241948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3646,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:46.988538+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:20:46.988538+0000","last_peered":"2023-10-21T05:20:46.988538+0000","last_clean":"2023-10-21T05:20:46.988538+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:20:46.988538+0000","last_undegraded":"2023-10-21T05:20:46.988538+0000","last_fullsized":"2023-10-21T05:20:46.988538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"147'596221","reported_seq":1176585,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:50.493240+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:20:50.493240+0000","last_peered":"2023-10-21T05:20:50.493240+0000","last_clean":"2023-10-21T05:20:50.493240+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:20:50.493240+0000","last_undegraded":"2023-10-21T05:20:50.493240+0000","last_fullsized":"2023-10-21T05:20:50.493240+0000","mapping_epoch":14,"log_start":"145'586154","ondisk_log_start":"145'586154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6337,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 869s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7881187328,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":584923,"num_read_kb":2310536,"num_write":585748,"num_write_kb":2342992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7252,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:46.980002+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:20:46.980002+0000","last_peered":"2023-10-21T05:20:46.980002+0000","last_clean":"2023-10-21T05:20:46.980002+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:20:46.980002+0000","last_undegraded":"2023-10-21T05:20:46.980002+0000","last_fullsized":"2023-10-21T05:20:46.980002+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'545020","reported_seq":1081507,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:51.210556+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:20:51.210556+0000","last_peered":"2023-10-21T05:20:51.210556+0000","last_clean":"2023-10-21T05:20:51.210556+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:20:51.210556+0000","last_undegraded":"2023-10-21T05:20:51.210556+0000","last_fullsized":"2023-10-21T05:20:51.210556+0000","mapping_epoch":14,"log_start":"145'534958","ondisk_log_start":"145'534958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1620,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 179s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7256838163,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":540784,"num_read_kb":2136188,"num_write":538831,"num_write_kb":2155318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'7471","reported_seq":10914,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:50.503609+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:20:50.503609+0000","last_peered":"2023-10-21T05:20:50.503609+0000","last_clean":"2023-10-21T05:20:50.503609+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:20:50.503609+0000","last_undegraded":"2023-10-21T05:20:50.503609+0000","last_fullsized":"2023-10-21T05:20:50.503609+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":325,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.112689+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:20:26.112689+0000","last_peered":"2023-10-21T05:20:26.112689+0000","last_clean":"2023-10-21T05:20:26.112689+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:20:26.112689+0000","last_undegraded":"2023-10-21T05:20:26.112689+0000","last_fullsized":"2023-10-21T05:20:26.112689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30136332307,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2243839,"num_read_kb":8862284,"num_write":2239127,"num_write_kb":8956502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18343501824,"data_stored":37947318310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40250,"ondisk_log_size":40250,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":46938,"ondisk_log_size":46938,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739345,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10067628,"kb_used_data":8957860,"kb_used_omap":2691,"kb_used_meta":1107068,"kb_avail":84304212,"statfs":{"total":96636764160,"available":86327513088,"internally_reserved":0,"allocated":9172848640,"data_stored":18973745097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1133637777},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":1.236,"15min":1.0529999999999999},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.0950000000000002,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.284,"15min":1.0129999999999999},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.863,"5min":27.251000000000001,"15min":27.251000000000001},"last":2.2650000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86699999999999999,"15min":0.97699999999999998},"min":{"1min":0.222,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":3.254,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0589999999999999,"15min":1.1080000000000001},"min":{"1min":0.26500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.2229999999999999,"5min":5.1660000000000004,"15min":10.253},"last":0.53100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739344,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5942204,"kb_used_data":4398964,"kb_used_omap":1192,"kb_used_meta":1542039,"kb_avail":88429636,"statfs":{"total":96636764160,"available":90551947264,"internally_reserved":0,"allocated":4504539136,"data_stored":9315602333,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1579048883},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,9,1,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.752,"15min":0.81799999999999995},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":3.3919999999999999,"5min":4.1050000000000004,"15min":10.711},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":0.88900000000000001},"min":{"1min":0.24199999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":3.3519999999999999,"5min":4.1879999999999997,"15min":10.512},"last":0.32800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89500000000000002,"15min":0.96699999999999997},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.19900000000000001},"max":{"1min":3.4279999999999999,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.371},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.84399999999999997,"15min":0.95999999999999996},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.9330000000000001,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6170640,"kb_used_data":4560648,"kb_used_omap":1353,"kb_used_meta":1608630,"kb_avail":88201200,"statfs":{"total":96636764160,"available":90318028800,"internally_reserved":0,"allocated":4670103552,"data_stored":9659512948,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1647237494},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,2,4,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.81299999999999994,"15min":0.82499999999999996},"min":{"1min":0.23200000000000001,"5min":0.189,"15min":0.187},"max":{"1min":4.5570000000000004,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.89300000000000002,"15min":0.90600000000000003},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":5.968,"5min":5.968,"15min":6.4050000000000002},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:20:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.93100000000000005,"15min":0.93999999999999995},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":2.3580000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.008,"15min":0.98799999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":4.702,"5min":5.0270000000000001,"15min":5.133},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4668657664,"data_stored":9658703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4502810624,"data_stored":9314672659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9172033536,"data_stored":18973941779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:54.187 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:54.187 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:54.188 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:59.070 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.067+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:20:59.077 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.069+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:20:59.077 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.069+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:20:59.078 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.069+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:20:59.085 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.083+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:20:59.085 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.083+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:20:59.085 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.083+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:20:59.086 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:20:59.083+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:21:07.949 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:07.945+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:21:07.949 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:07.945+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:21:07.950 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:07.945+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:21:10.309 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.305+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:21:10.310 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.305+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:21:10.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:21:10.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.312+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:21:10.316 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.312+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:21:10.326 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:21:10.326 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:21:10.327 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:21:10.327 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:21:10.327 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:21:10.328 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:21:10.328 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:21:10.328 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:21:10.328 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:21:10.323+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:21:14.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:14.732 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:21:14.733 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:21:14.754 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2592,"stamp":"2023-10-21T05:21:13.928752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30169319165,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2301102,"num_read_kb":8934390,"num_write":2248166,"num_write_kb":8990439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87498,"ondisk_log_size":87498,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22362792,"kb_used_data":17980560,"kb_used_omap":5237,"kb_used_meta":4376970,"kb_avail":260752728,"statfs":{"total":289910292480,"available":267010793472,"internally_reserved":0,"allocated":18412093440,"data_stored":38013462490,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":4482017722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,11,10,5,17,18,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15925248,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3933,"num_read_kb":15644,"num_write":3874,"num_write_kb":15496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002119"},"pg_stats":[{"pgid":"2.7","version":"147'7543","reported_seq":11063,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:11.074460+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:21:11.074460+0000","last_peered":"2023-10-21T05:21:11.074460+0000","last_clean":"2023-10-21T05:21:11.074460+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:21:11.074460+0000","last_undegraded":"2023-10-21T05:21:11.074460+0000","last_fullsized":"2023-10-21T05:21:11.074460+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'3860","reported_seq":7661,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.313727+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:21:10.313727+0000","last_peered":"2023-10-21T05:21:10.313727+0000","last_clean":"2023-10-21T05:21:10.313727+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:21:10.313727+0000","last_undegraded":"2023-10-21T05:21:10.313727+0000","last_fullsized":"2023-10-21T05:21:10.313727+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'11257","reported_seq":16396,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:11.082384+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:21:11.082384+0000","last_peered":"2023-10-21T05:21:11.082384+0000","last_clean":"2023-10-21T05:21:11.082384+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:21:11.082384+0000","last_undegraded":"2023-10-21T05:21:11.082384+0000","last_fullsized":"2023-10-21T05:21:11.082384+0000","mapping_epoch":11,"log_start":"53'1200","ondisk_log_start":"53'1200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'7583","reported_seq":12117,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:11.074513+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:21:11.074513+0000","last_peered":"2023-10-21T05:21:11.074513+0000","last_clean":"2023-10-21T05:21:11.074513+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:21:11.074513+0000","last_undegraded":"2023-10-21T05:21:11.074513+0000","last_fullsized":"2023-10-21T05:21:11.074513+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"147'566159","reported_seq":1119095,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:10.637178+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:21:10.637178+0000","last_peered":"2023-10-21T05:21:10.637178+0000","last_clean":"2023-10-21T05:21:10.637178+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:21:10.637178+0000","last_undegraded":"2023-10-21T05:21:10.637178+0000","last_fullsized":"2023-10-21T05:21:10.637178+0000","mapping_epoch":14,"log_start":"146'556062","ondisk_log_start":"146'556062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7461117952,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":557239,"num_read_kb":2200876,"num_write":556004,"num_write_kb":2224016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'15261","reported_seq":22256,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:09.558544+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:21:09.558544+0000","last_peered":"2023-10-21T05:21:09.558544+0000","last_clean":"2023-10-21T05:21:09.558544+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:21:09.558544+0000","last_undegraded":"2023-10-21T05:21:09.558544+0000","last_fullsized":"2023-10-21T05:21:09.558544+0000","mapping_epoch":11,"log_start":"91'5200","ondisk_log_start":"91'5200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'572706","reported_seq":1133086,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:11.262662+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:21:11.262662+0000","last_peered":"2023-10-21T05:21:11.262662+0000","last_clean":"2023-10-21T05:21:11.262662+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:21:11.262662+0000","last_undegraded":"2023-10-21T05:21:11.262662+0000","last_fullsized":"2023-10-21T05:21:11.262662+0000","mapping_epoch":14,"log_start":"146'562661","ondisk_log_start":"146'562661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7553257472,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":564722,"num_read_kb":2229908,"num_write":562465,"num_write_kb":2249860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3654,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:07.758325+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:21:07.758325+0000","last_peered":"2023-10-21T05:21:07.758325+0000","last_clean":"2023-10-21T05:21:07.758325+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:21:07.758325+0000","last_undegraded":"2023-10-21T05:21:07.758325+0000","last_fullsized":"2023-10-21T05:21:07.758325+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"147'598410","reported_seq":1180809,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:10.617309+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:21:10.617309+0000","last_peered":"2023-10-21T05:21:10.617309+0000","last_clean":"2023-10-21T05:21:10.617309+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:21:10.617309+0000","last_undegraded":"2023-10-21T05:21:10.617309+0000","last_fullsized":"2023-10-21T05:21:10.617309+0000","mapping_epoch":14,"log_start":"146'588354","ondisk_log_start":"146'588354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6475,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 889s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7889719296,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":586950,"num_read_kb":2318612,"num_write":587825,"num_write_kb":2351300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7268,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:07.550291+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:21:07.550291+0000","last_peered":"2023-10-21T05:21:07.550291+0000","last_clean":"2023-10-21T05:21:07.550291+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:21:07.550291+0000","last_undegraded":"2023-10-21T05:21:07.550291+0000","last_fullsized":"2023-10-21T05:21:07.550291+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'547031","reported_seq":1085481,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:11.294534+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:21:11.294534+0000","last_peered":"2023-10-21T05:21:11.294534+0000","last_clean":"2023-10-21T05:21:11.294534+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:21:11.294534+0000","last_undegraded":"2023-10-21T05:21:11.294534+0000","last_fullsized":"2023-10-21T05:21:11.294534+0000","mapping_epoch":14,"log_start":"146'536958","ondisk_log_start":"146'536958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1776,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 199s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7264628755,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":542732,"num_read_kb":2143924,"num_write":540714,"num_write_kb":2162850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'7501","reported_seq":10952,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:11.073608+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:21:11.073608+0000","last_peered":"2023-10-21T05:21:11.073608+0000","last_clean":"2023-10-21T05:21:11.073608+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:21:11.073608+0000","last_undegraded":"2023-10-21T05:21:11.073608+0000","last_fullsized":"2023-10-21T05:21:11.073608+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":325,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.112689+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:20:26.112689+0000","last_peered":"2023-10-21T05:20:26.112689+0000","last_clean":"2023-10-21T05:20:26.112689+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:20:26.112689+0000","last_undegraded":"2023-10-21T05:20:26.112689+0000","last_fullsized":"2023-10-21T05:20:26.112689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30168723475,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2251643,"num_read_kb":8893320,"num_write":2247008,"num_write_kb":8988026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18407989248,"data_stored":38011805734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40271,"ondisk_log_size":40271,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47128,"ondisk_log_size":47128,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739349,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10135272,"kb_used_data":8989344,"kb_used_omap":2691,"kb_used_meta":1143228,"kb_avail":84236568,"statfs":{"total":96636764160,"available":86258245632,"internally_reserved":0,"allocated":9205088256,"data_stored":19005984713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1170665617},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":1.236,"15min":1.0529999999999999},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.0950000000000002,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.284,"15min":1.0129999999999999},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.863,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.36099999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86699999999999999,"15min":0.97699999999999998},"min":{"1min":0.222,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":3.254,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0589999999999999,"15min":1.1080000000000001},"min":{"1min":0.26500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.2229999999999999,"5min":5.1660000000000004,"15min":10.253},"last":0.378}]}]},{"osd":1,"up_from":8,"seq":34359739348,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5960428,"kb_used_data":4414436,"kb_used_omap":1192,"kb_used_meta":1544791,"kb_avail":88411412,"statfs":{"total":96636764160,"available":90533285888,"internally_reserved":0,"allocated":4520382464,"data_stored":9331445661,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1581866931},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,7,4,3,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.752,"15min":0.81799999999999995},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":3.3919999999999999,"5min":4.1050000000000004,"15min":10.711},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":0.88900000000000001},"min":{"1min":0.24199999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":3.3519999999999999,"5min":4.1879999999999997,"15min":10.512},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:20:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89500000000000002,"15min":0.96699999999999997},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.19900000000000001},"max":{"1min":3.4279999999999999,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.84399999999999997,"15min":0.95999999999999996},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.9330000000000001,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6267092,"kb_used_data":4576780,"kb_used_omap":1353,"kb_used_meta":1688950,"kb_avail":88104748,"statfs":{"total":96636764160,"available":90219261952,"internally_reserved":0,"allocated":4686622720,"data_stored":9676032116,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1729485174},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,5,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.81299999999999994,"15min":0.82499999999999996},"min":{"1min":0.23200000000000001,"5min":0.189,"15min":0.187},"max":{"1min":4.5570000000000004,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.89300000000000002,"15min":0.90600000000000003},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":5.968,"5min":5.968,"15min":6.4050000000000002},"last":0.495}]},{"osd":2,"last update":"Sat Oct 21 05:20:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.93100000000000005,"15min":0.93999999999999995},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":2.3580000000000001,"5min":5.141,"15min":5.3179999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.008,"15min":0.98799999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20100000000000001},"max":{"1min":4.702,"5min":5.0270000000000001,"15min":5.133},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4685135872,"data_stored":9675182080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4518633472,"data_stored":9330495507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9204219904,"data_stored":19006128147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:14.756 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:14.757 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:14.757 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:17.270 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:17.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:21:17.270 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:17.267+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:21:17.275 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:17.273+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:21:17.702 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:17.697+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:21:17.703 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:17.697+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:21:27.644 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:27.643+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:21:34.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:21:35.271 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:21:35.290 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2602,"stamp":"2023-10-21T05:21:33.932255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30203094781,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2309235,"num_read_kb":8966822,"num_write":2256386,"num_write_kb":9023319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87606,"ondisk_log_size":87606,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22726560,"kb_used_data":18045188,"kb_used_omap":5237,"kb_used_meta":4676106,"kb_avail":260388960,"statfs":{"total":289910292480,"available":266638295040,"internally_reserved":0,"allocated":18478272512,"data_stored":38079641562,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":4788332986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,8,5,13,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16945152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3974,"num_read_kb":15832,"num_write":4123,"num_write_kb":16492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002156"},"pg_stats":[{"pgid":"2.7","version":"147'7567","reported_seq":11095,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:30.353237+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:21:30.353237+0000","last_peered":"2023-10-21T05:21:30.353237+0000","last_clean":"2023-10-21T05:21:30.353237+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:21:30.353237+0000","last_undegraded":"2023-10-21T05:21:30.353237+0000","last_fullsized":"2023-10-21T05:21:30.353237+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'3872","reported_seq":7678,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:29.723853+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:21:29.723853+0000","last_peered":"2023-10-21T05:21:29.723853+0000","last_clean":"2023-10-21T05:21:29.723853+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:21:29.723853+0000","last_undegraded":"2023-10-21T05:21:29.723853+0000","last_fullsized":"2023-10-21T05:21:29.723853+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'11293","reported_seq":16444,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:30.372507+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:21:30.372507+0000","last_peered":"2023-10-21T05:21:30.372507+0000","last_clean":"2023-10-21T05:21:30.372507+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:21:30.372507+0000","last_undegraded":"2023-10-21T05:21:30.372507+0000","last_fullsized":"2023-10-21T05:21:30.372507+0000","mapping_epoch":11,"log_start":"53'1200","ondisk_log_start":"53'1200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'7607","reported_seq":12149,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:30.356976+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:21:30.356976+0000","last_peered":"2023-10-21T05:21:30.356976+0000","last_clean":"2023-10-21T05:21:30.356976+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:21:30.356976+0000","last_undegraded":"2023-10-21T05:21:30.356976+0000","last_fullsized":"2023-10-21T05:21:30.356976+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"147'568225","reported_seq":1123161,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:30.838039+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:21:30.838039+0000","last_peered":"2023-10-21T05:21:30.838039+0000","last_clean":"2023-10-21T05:21:30.838039+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:21:30.838039+0000","last_undegraded":"2023-10-21T05:21:30.838039+0000","last_fullsized":"2023-10-21T05:21:30.838039+0000","mapping_epoch":14,"log_start":"146'558162","ondisk_log_start":"146'558162","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7469584384,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":559240,"num_read_kb":2208856,"num_write":558070,"num_write_kb":2232280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'15310","reported_seq":22321,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:30.345945+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:21:30.345945+0000","last_peered":"2023-10-21T05:21:30.345945+0000","last_clean":"2023-10-21T05:21:30.345945+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:21:30.345945+0000","last_undegraded":"2023-10-21T05:21:30.345945+0000","last_fullsized":"2023-10-21T05:21:30.345945+0000","mapping_epoch":11,"log_start":"93'5300","ondisk_log_start":"93'5300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'574760","reported_seq":1137273,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:31.485999+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:21:31.485999+0000","last_peered":"2023-10-21T05:21:31.485999+0000","last_clean":"2023-10-21T05:21:31.485999+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:21:31.485999+0000","last_undegraded":"2023-10-21T05:21:31.485999+0000","last_fullsized":"2023-10-21T05:21:31.485999+0000","mapping_epoch":14,"log_start":"146'564661","ondisk_log_start":"146'564661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7561691136,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":566854,"num_read_kb":2238404,"num_write":564519,"num_write_kb":2258076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3663,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:27.958489+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:21:27.958489+0000","last_peered":"2023-10-21T05:21:27.958489+0000","last_clean":"2023-10-21T05:21:27.958489+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:21:27.958489+0000","last_undegraded":"2023-10-21T05:21:27.958489+0000","last_fullsized":"2023-10-21T05:21:27.958489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"147'600611","reported_seq":1185096,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:30.819030+0000","last_change":"2023-10-21T05:06:20.924145+0000","last_active":"2023-10-21T05:21:30.819030+0000","last_peered":"2023-10-21T05:21:30.819030+0000","last_clean":"2023-10-21T05:21:30.819030+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:21:30.819030+0000","last_undegraded":"2023-10-21T05:21:30.819030+0000","last_fullsized":"2023-10-21T05:21:30.819030+0000","mapping_epoch":14,"log_start":"146'590554","ondisk_log_start":"146'590554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6585,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 909s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7898419200,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":589019,"num_read_kb":2326856,"num_write":589936,"num_write_kb":2359744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7284,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:27.937306+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:21:27.937306+0000","last_peered":"2023-10-21T05:21:27.937306+0000","last_clean":"2023-10-21T05:21:27.937306+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:21:27.937306+0000","last_undegraded":"2023-10-21T05:21:27.937306+0000","last_fullsized":"2023-10-21T05:21:27.937306+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'549149","reported_seq":1089540,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:31.484310+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:21:31.484310+0000","last_peered":"2023-10-21T05:21:31.484310+0000","last_clean":"2023-10-21T05:21:31.484310+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:21:31.484310+0000","last_undegraded":"2023-10-21T05:21:31.484310+0000","last_fullsized":"2023-10-21T05:21:31.484310+0000","mapping_epoch":14,"log_start":"146'539058","ondisk_log_start":"146'539058","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":1932,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 220s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7272804371,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":544663,"num_read_kb":2151636,"num_write":542703,"num_write_kb":2170806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'7525","reported_seq":10984,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:30.364716+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:21:30.364716+0000","last_peered":"2023-10-21T05:21:30.364716+0000","last_clean":"2023-10-21T05:21:30.364716+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:21:30.364716+0000","last_undegraded":"2023-10-21T05:21:30.364716+0000","last_fullsized":"2023-10-21T05:21:30.364716+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7525,"log_dups_size":0,"ondisk_log_size":7525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":325,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-21T05:20:26.112689+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:20:26.112689+0000","last_peered":"2023-10-21T05:20:26.112689+0000","last_clean":"2023-10-21T05:20:26.112689+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:20:26.112689+0000","last_undegraded":"2023-10-21T05:20:26.112689+0000","last_fullsized":"2023-10-21T05:20:26.112689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30202499091,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2259776,"num_read_kb":8925752,"num_write":2255228,"num_write_kb":9020906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18475122688,"data_stored":38078939174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40310,"ondisk_log_size":40310,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47197,"ondisk_log_size":47197,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739353,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10288204,"kb_used_data":9021508,"kb_used_omap":2691,"kb_used_meta":1263996,"kb_avail":84083636,"statfs":{"total":96636764160,"available":86101643264,"internally_reserved":0,"allocated":9238024192,"data_stored":19038920649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1294332049},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.2450000000000001,"15min":1.0189999999999999},"min":{"1min":0.253,"5min":0.186,"15min":0.186},"max":{"1min":3.6669999999999998,"5min":27.184999999999999,"15min":27.184999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.1659999999999999,"15min":0.98899999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.2650000000000001,"5min":27.251000000000001,"15min":27.251000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.90300000000000002,"15min":0.95199999999999996},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":3.7080000000000002,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.29299999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.7370000000000001,"5min":5.1660000000000004,"15min":10.253},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739352,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6066592,"kb_used_data":4430424,"kb_used_omap":1192,"kb_used_meta":1634967,"kb_avail":88305248,"statfs":{"total":96636764160,"available":90424573952,"internally_reserved":0,"allocated":4536754176,"data_stored":9347817373,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1674207155},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.77100000000000002},"min":{"1min":0.219,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":3.5270000000000001,"5min":4.1050000000000004,"15min":6.2140000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.25800000000000001,"5min":0.24199999999999999,"15min":0.20699999999999999},"max":{"1min":2.1850000000000001,"5min":4.1879999999999997,"15min":6.0579999999999998},"last":0.44700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.81999999999999995,"15min":0.91100000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.19900000000000001},"max":{"1min":2.9220000000000002,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.75700000000000001,"15min":0.90200000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.7810000000000001,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":1.3640000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6371764,"kb_used_data":4593256,"kb_used_omap":1353,"kb_used_meta":1777142,"kb_avail":88000076,"statfs":{"total":96636764160,"available":90112077824,"internally_reserved":0,"allocated":4703494144,"data_stored":9692903540,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1819793782},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.746,"15min":0.81100000000000005},"min":{"1min":0.27100000000000002,"5min":0.189,"15min":0.187},"max":{"1min":2.3109999999999999,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.79600000000000004,"15min":0.88},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":2.3500000000000001,"5min":5.968,"15min":6.4050000000000002},"last":0.46600000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.877,"15min":0.95099999999999996},"min":{"1min":0.248,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.95499999999999996,"15min":0.96699999999999997},"min":{"1min":0.25700000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":6.1779999999999999,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":1.206}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4702236672,"data_stored":9692282880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4535177216,"data_stored":9347039251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9237708800,"data_stored":19039617043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:35.292 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:35.292 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:35.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:21:35.808 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:21:35.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:21:36.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-21T05:21:36.227+0000 7f04b258c640 0 --2- 172.21.15.40:0/3189867474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0490003610 0x7f0490005a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T05:21:36.573 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:21:36.598 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:40.330 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:40.328+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:21:40.331 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:40.328+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:21:40.331 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:40.328+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:21:50.972 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:50.966+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:21:50.972 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:21:50.966+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:21:56.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:57.679 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:21:57.680 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:21:57.702 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2614,"stamp":"2023-10-21T05:21:55.936328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30237304573,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2317586,"num_read_kb":9000078,"num_write":2264716,"num_write_kb":9056639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87689,"ondisk_log_size":87689,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23170020,"kb_used_data":18111944,"kb_used_omap":5237,"kb_used_meta":5052810,"kb_avail":259945500,"statfs":{"total":289910292480,"available":266184192000,"internally_reserved":0,"allocated":18546630656,"data_stored":38147976021,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":5174077882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,2,6,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13086720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3205,"num_read_kb":12768,"num_write":3196,"num_write_kb":12784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001705"},"pg_stats":[{"pgid":"2.7","version":"148'7597","reported_seq":11134,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.989498+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:21:49.989498+0000","last_peered":"2023-10-21T05:21:49.989498+0000","last_clean":"2023-10-21T05:21:49.989498+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:21:49.989498+0000","last_undegraded":"2023-10-21T05:21:49.989498+0000","last_fullsized":"2023-10-21T05:21:49.989498+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"148'3887","reported_seq":7698,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.547035+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:21:49.547035+0000","last_peered":"2023-10-21T05:21:49.547035+0000","last_clean":"2023-10-21T05:21:49.547035+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:21:49.547035+0000","last_undegraded":"2023-10-21T05:21:49.547035+0000","last_fullsized":"2023-10-21T05:21:49.547035+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"148'11338","reported_seq":16503,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.992029+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:21:49.992029+0000","last_peered":"2023-10-21T05:21:49.992029+0000","last_clean":"2023-10-21T05:21:49.992029+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:21:49.992029+0000","last_undegraded":"2023-10-21T05:21:49.992029+0000","last_fullsized":"2023-10-21T05:21:49.992029+0000","mapping_epoch":11,"log_start":"55'1300","ondisk_log_start":"55'1300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"148'7637","reported_seq":12189,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.988845+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:21:49.988845+0000","last_peered":"2023-10-21T05:21:49.988845+0000","last_clean":"2023-10-21T05:21:49.988845+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:21:49.988845+0000","last_undegraded":"2023-10-21T05:21:49.988845+0000","last_fullsized":"2023-10-21T05:21:49.988845+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7637,"log_dups_size":0,"ondisk_log_size":7637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'570237","reported_seq":1127226,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.936611+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:21:50.936611+0000","last_peered":"2023-10-21T05:21:50.936611+0000","last_clean":"2023-10-21T05:21:50.936611+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:21:50.936611+0000","last_undegraded":"2023-10-21T05:21:50.936611+0000","last_fullsized":"2023-10-21T05:21:50.936611+0000","mapping_epoch":14,"log_start":"146'560162","ondisk_log_start":"146'560162","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7477821440,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":561293,"num_read_kb":2217036,"num_write":560082,"num_write_kb":2240328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'15394","reported_seq":22423,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.514558+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:21:50.514558+0000","last_peered":"2023-10-21T05:21:50.514558+0000","last_clean":"2023-10-21T05:21:50.514558+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:21:50.514558+0000","last_undegraded":"2023-10-21T05:21:50.514558+0000","last_fullsized":"2023-10-21T05:21:50.514558+0000","mapping_epoch":11,"log_start":"93'5300","ondisk_log_start":"93'5300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"148'576874","reported_seq":1141528,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:51.637964+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:21:51.637964+0000","last_peered":"2023-10-21T05:21:51.637964+0000","last_clean":"2023-10-21T05:21:51.637964+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:21:51.637964+0000","last_undegraded":"2023-10-21T05:21:51.637964+0000","last_fullsized":"2023-10-21T05:21:51.637964+0000","mapping_epoch":14,"log_start":"146'566861","ondisk_log_start":"146'566861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7570391040,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":568996,"num_read_kb":2246920,"num_write":566633,"num_write_kb":2266532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3672,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:48.417079+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:21:48.417079+0000","last_peered":"2023-10-21T05:21:48.417079+0000","last_clean":"2023-10-21T05:21:48.417079+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:21:48.417079+0000","last_undegraded":"2023-10-21T05:21:48.417079+0000","last_fullsized":"2023-10-21T05:21:48.417079+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'602831","reported_seq":1189517,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:50.936227+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:21:50.936227+0000","last_peered":"2023-10-21T05:21:50.936227+0000","last_clean":"2023-10-21T05:21:50.936227+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:21:50.936227+0000","last_undegraded":"2023-10-21T05:21:50.936227+0000","last_fullsized":"2023-10-21T05:21:50.936227+0000","mapping_epoch":14,"log_start":"146'592754","ondisk_log_start":"146'592754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7907381248,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":591208,"num_read_kb":2335560,"num_write":592117,"num_write_kb":2368468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7301,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:21:48.418446+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:21:48.418446+0000","last_peered":"2023-10-21T05:21:48.418446+0000","last_clean":"2023-10-21T05:21:48.418446+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:21:48.418446+0000","last_undegraded":"2023-10-21T05:21:48.418446+0000","last_fullsized":"2023-10-21T05:21:48.418446+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"148'551252","reported_seq":1093623,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:51.645457+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:21:51.645457+0000","last_peered":"2023-10-21T05:21:51.645457+0000","last_clean":"2023-10-21T05:21:51.645457+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:21:51.645457+0000","last_undegraded":"2023-10-21T05:21:51.645457+0000","last_fullsized":"2023-10-21T05:21:51.645457+0000","mapping_epoch":14,"log_start":"146'541158","ondisk_log_start":"146'541158","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2031,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 240s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7281115155,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":546630,"num_read_kb":2159492,"num_write":544726,"num_write_kb":2178898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"148'7555","reported_seq":11024,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.998492+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:21:49.998492+0000","last_peered":"2023-10-21T05:21:49.998492+0000","last_clean":"2023-10-21T05:21:49.998492+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:21:49.998492+0000","last_undegraded":"2023-10-21T05:21:49.998492+0000","last_fullsized":"2023-10-21T05:21:49.998492+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7555,"log_dups_size":0,"ondisk_log_size":7555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":326,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.526497+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:20:39.526497+0000","last_peered":"2023-10-21T05:20:39.526497+0000","last_clean":"2023-10-21T05:20:39.526497+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:20:39.526497+0000","last_undegraded":"2023-10-21T05:20:39.526497+0000","last_fullsized":"2023-10-21T05:20:39.526497+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30236708883,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2268127,"num_read_kb":8959008,"num_write":2263558,"num_write_kb":9054226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18543378432,"data_stored":38147194918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47331,"ondisk_log_size":47331,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739357,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10527432,"kb_used_data":9054720,"kb_used_omap":2691,"kb_used_meta":1470012,"kb_avail":83844408,"statfs":{"total":96636764160,"available":85856673792,"internally_reserved":0,"allocated":9272033280,"data_stored":19072921842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1505292433},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.2450000000000001,"15min":1.0189999999999999},"min":{"1min":0.253,"5min":0.186,"15min":0.186},"max":{"1min":3.6669999999999998,"5min":27.184999999999999,"15min":27.184999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.1659999999999999,"15min":0.98899999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.2650000000000001,"5min":27.251000000000001,"15min":27.251000000000001},"last":1.0940000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.90300000000000002,"15min":0.95199999999999996},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":3.7080000000000002,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.29299999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.7370000000000001,"5min":5.1660000000000004,"15min":10.253},"last":0.29099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739356,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6177576,"kb_used_data":4447072,"kb_used_omap":1192,"kb_used_meta":1729303,"kb_avail":88194264,"statfs":{"total":96636764160,"available":90310926336,"internally_reserved":0,"allocated":4553801728,"data_stored":9364857030,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1770807219},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,4,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.77100000000000002},"min":{"1min":0.219,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":3.5270000000000001,"5min":4.1050000000000004,"15min":6.2140000000000004},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.25800000000000001,"5min":0.24199999999999999,"15min":0.20699999999999999},"max":{"1min":2.1850000000000001,"5min":4.1879999999999997,"15min":6.0579999999999998},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.81999999999999995,"15min":0.91100000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.19900000000000001},"max":{"1min":2.9220000000000002,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.496},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.75700000000000001,"15min":0.90200000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.7810000000000001,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6465012,"kb_used_data":4610152,"kb_used_omap":1353,"kb_used_meta":1853494,"kb_avail":87906828,"statfs":{"total":96636764160,"available":90016591872,"internally_reserved":0,"allocated":4720795648,"data_stored":9710197149,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1897978230},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.746,"15min":0.81100000000000005},"min":{"1min":0.27100000000000002,"5min":0.189,"15min":0.187},"max":{"1min":2.3109999999999999,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.79600000000000004,"15min":0.88},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":2.3500000000000001,"5min":5.968,"15min":6.4050000000000002},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.877,"15min":0.95099999999999996},"min":{"1min":0.248,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.95499999999999996,"15min":0.96699999999999997},"min":{"1min":0.25700000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":6.1779999999999999,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4719415296,"data_stored":9709461504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4552130560,"data_stored":9363992595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9271832576,"data_stored":19073740819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:57.704 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:57.705 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:06.859 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:22:06.859 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:22:06.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:22:06.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:22:06.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:22:06.860 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:22:06.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:22:06.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:22:06.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:22:06.861 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:06.856+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:22:17.706 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:18.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:22:18.247 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:22:18.266 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2625,"stamp":"2023-10-21T05:22:17.940241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30281094909,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2327985,"num_read_kb":9041554,"num_write":2275381,"num_write_kb":9099299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87802,"ondisk_log_size":87802,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23669684,"kb_used_data":18197080,"kb_used_omap":5237,"kb_used_meta":5467338,"kb_avail":259445836,"statfs":{"total":289910292480,"available":265672536064,"internally_reserved":0,"allocated":18633809920,"data_stored":38235155285,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5363270,"internal_metadata":5598554554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,2,9,14,8,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26595328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6261,"num_read_kb":24964,"num_write":6482,"num_write_kb":25928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002135"},"pg_stats":[{"pgid":"2.7","version":"148'7629","reported_seq":11176,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.279540+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:22:15.279540+0000","last_peered":"2023-10-21T05:22:15.279540+0000","last_clean":"2023-10-21T05:22:15.279540+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:22:15.279540+0000","last_undegraded":"2023-10-21T05:22:15.279540+0000","last_fullsized":"2023-10-21T05:22:15.279540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"148'3903","reported_seq":7719,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:14.016623+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:22:14.016623+0000","last_peered":"2023-10-21T05:22:14.016623+0000","last_clean":"2023-10-21T05:22:14.016623+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:22:14.016623+0000","last_undegraded":"2023-10-21T05:22:14.016623+0000","last_fullsized":"2023-10-21T05:22:14.016623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"148'11386","reported_seq":16566,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.289409+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:22:15.289409+0000","last_peered":"2023-10-21T05:22:15.289409+0000","last_clean":"2023-10-21T05:22:15.289409+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:22:15.289409+0000","last_undegraded":"2023-10-21T05:22:15.289409+0000","last_fullsized":"2023-10-21T05:22:15.289409+0000","mapping_epoch":11,"log_start":"55'1300","ondisk_log_start":"55'1300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"148'7669","reported_seq":12231,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.279340+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:22:15.279340+0000","last_peered":"2023-10-21T05:22:15.279340+0000","last_clean":"2023-10-21T05:22:15.279340+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:22:15.279340+0000","last_undegraded":"2023-10-21T05:22:15.279340+0000","last_fullsized":"2023-10-21T05:22:15.279340+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'572826","reported_seq":1132399,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:16.188801+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:22:16.188801+0000","last_peered":"2023-10-21T05:22:16.188801+0000","last_clean":"2023-10-21T05:22:16.188801+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:22:16.188801+0000","last_undegraded":"2023-10-21T05:22:16.188801+0000","last_fullsized":"2023-10-21T05:22:16.188801+0000","mapping_epoch":14,"log_start":"146'562762","ondisk_log_start":"146'562762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7488434176,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":563874,"num_read_kb":2227316,"num_write":562671,"num_write_kb":2250684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'15474","reported_seq":22520,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.984489+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:22:15.984489+0000","last_peered":"2023-10-21T05:22:15.984489+0000","last_clean":"2023-10-21T05:22:15.984489+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:22:15.984489+0000","last_undegraded":"2023-10-21T05:22:15.984489+0000","last_fullsized":"2023-10-21T05:22:15.984489+0000","mapping_epoch":11,"log_start":"93'5400","ondisk_log_start":"93'5400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"148'579529","reported_seq":1146829,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:16.936690+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:22:16.936690+0000","last_peered":"2023-10-21T05:22:16.936690+0000","last_clean":"2023-10-21T05:22:16.936690+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:22:16.936690+0000","last_undegraded":"2023-10-21T05:22:16.936690+0000","last_fullsized":"2023-10-21T05:22:16.936690+0000","mapping_epoch":14,"log_start":"146'569461","ondisk_log_start":"146'569461","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7581298688,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":571640,"num_read_kb":2257480,"num_write":569288,"num_write_kb":2277152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3682,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:14.028241+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:22:14.028241+0000","last_peered":"2023-10-21T05:22:14.028241+0000","last_clean":"2023-10-21T05:22:14.028241+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:22:14.028241+0000","last_undegraded":"2023-10-21T05:22:14.028241+0000","last_fullsized":"2023-10-21T05:22:14.028241+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'605653","reported_seq":1194981,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:16.177332+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:22:16.177332+0000","last_peered":"2023-10-21T05:22:16.177332+0000","last_clean":"2023-10-21T05:22:16.177332+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:22:16.177332+0000","last_undegraded":"2023-10-21T05:22:16.177332+0000","last_fullsized":"2023-10-21T05:22:16.177332+0000","mapping_epoch":14,"log_start":"147'595554","ondisk_log_start":"147'595554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7918989312,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":593854,"num_read_kb":2346112,"num_write":594939,"num_write_kb":2379756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7322,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:14.025179+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:22:14.025179+0000","last_peered":"2023-10-21T05:22:14.025179+0000","last_clean":"2023-10-21T05:22:14.025179+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:22:14.025179+0000","last_undegraded":"2023-10-21T05:22:14.025179+0000","last_fullsized":"2023-10-21T05:22:14.025179+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"148'553959","reported_seq":1098873,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:16.932625+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:22:16.932625+0000","last_peered":"2023-10-21T05:22:16.932625+0000","last_clean":"2023-10-21T05:22:16.932625+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:22:16.932625+0000","last_undegraded":"2023-10-21T05:22:16.932625+0000","last_fullsized":"2023-10-21T05:22:16.932625+0000","mapping_epoch":14,"log_start":"147'543958","ondisk_log_start":"147'543958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2166,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7291777043,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":549158,"num_read_kb":2169576,"num_write":547325,"num_write_kb":2189294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"148'7587","reported_seq":11066,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.287227+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:22:15.287227+0000","last_peered":"2023-10-21T05:22:15.287227+0000","last_clean":"2023-10-21T05:22:15.287227+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:22:15.287227+0000","last_undegraded":"2023-10-21T05:22:15.287227+0000","last_fullsized":"2023-10-21T05:22:15.287227+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7587,"log_dups_size":0,"ondisk_log_size":7587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":326,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.526497+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:20:39.526497+0000","last_peered":"2023-10-21T05:20:39.526497+0000","last_clean":"2023-10-21T05:20:39.526497+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:20:39.526497+0000","last_undegraded":"2023-10-21T05:20:39.526497+0000","last_fullsized":"2023-10-21T05:20:39.526497+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30280499219,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2278526,"num_read_kb":9000484,"num_write":2274223,"num_write_kb":9096886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18630406144,"data_stored":38234222630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47471,"ondisk_log_size":47471,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739362,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10822312,"kb_used_data":9097248,"kb_used_omap":2691,"kb_used_meta":1722364,"kb_avail":83549528,"statfs":{"total":96636764160,"available":85554716672,"internally_reserved":0,"allocated":9315581952,"data_stored":19116470514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1763700881},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.2450000000000001,"15min":1.0189999999999999},"min":{"1min":0.253,"5min":0.186,"15min":0.186},"max":{"1min":3.6669999999999998,"5min":27.184999999999999,"15min":27.184999999999999},"last":1.679},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.1659999999999999,"15min":0.98899999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.2650000000000001,"5min":27.251000000000001,"15min":27.251000000000001},"last":2.2749999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.90300000000000002,"15min":0.95199999999999996},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":3.7080000000000002,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.29299999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.7370000000000001,"5min":5.1660000000000004,"15min":10.253},"last":1.712}]}]},{"osd":1,"up_from":8,"seq":34359739361,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6300972,"kb_used_data":4468004,"kb_used_omap":1192,"kb_used_meta":1831767,"kb_avail":88070868,"statfs":{"total":96636764160,"available":90184568832,"internally_reserved":0,"allocated":4575236096,"data_stored":9386291398,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1875730355},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,8,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.79700000000000004,"15min":0.77100000000000002},"min":{"1min":0.219,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":3.5270000000000001,"5min":4.1050000000000004,"15min":6.2140000000000004},"last":0.246},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.25800000000000001,"5min":0.24199999999999999,"15min":0.20699999999999999},"max":{"1min":2.1850000000000001,"5min":4.1879999999999997,"15min":6.0579999999999998},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:21:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.81999999999999995,"15min":0.91100000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.19900000000000001},"max":{"1min":2.9220000000000002,"5min":5.0339999999999998,"15min":10.167999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.75700000000000001,"15min":0.90200000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.7810000000000001,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.71799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6546400,"kb_used_data":4631828,"kb_used_omap":1353,"kb_used_meta":1913206,"kb_avail":87825440,"statfs":{"total":96636764160,"available":89933250560,"internally_reserved":0,"allocated":4742991872,"data_stored":9732393373,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1386122,"internal_metadata":1959123318},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,12],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.746,"15min":0.81100000000000005},"min":{"1min":0.27100000000000002,"5min":0.189,"15min":0.187},"max":{"1min":2.3109999999999999,"5min":4.9870000000000001,"15min":5.0339999999999998},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.79600000000000004,"15min":0.88},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":2.3500000000000001,"5min":5.968,"15min":6.4050000000000002},"last":0.871}]},{"osd":2,"last update":"Sat Oct 21 05:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.877,"15min":0.95099999999999996},"min":{"1min":0.248,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":6.1029999999999998,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.95499999999999996,"15min":0.96699999999999997},"min":{"1min":0.25700000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":6.1779999999999999,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.98799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4741574656,"data_stored":9731620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4573646848,"data_stored":9385508883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9315184640,"data_stored":19117092883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:18.268 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:18.268 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:19.079 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:22:19.079 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:22:19.080 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:22:19.080 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:22:19.080 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:22:19.080 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:22:19.081 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:22:19.081 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:22:19.081 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.076+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:22:19.877 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:22:19.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:22:19.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:22:19.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:22:19.878 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:22:19.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:22:19.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:22:19.879 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:22:19.880 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:22:19.880 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:19.874+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:22:21.818 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.812+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:22:21.840 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.831+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:22:21.841 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.831+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:22:21.841 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.836+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:22:21.841 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.836+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:22:21.842 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.836+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:22:21.842 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:21.836+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:22:25.942 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:22:25.939+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:22:27.910 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:27.906+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:22:35.891 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:22:35.888+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:22:35.891 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:22:35.888+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:22:38.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:38.874 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:22:38.875 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:22:38.900 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2635,"stamp":"2023-10-21T05:22:37.943557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30315902717,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2336488,"num_read_kb":9075458,"num_write":2283870,"num_write_kb":9133255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87726,"ondisk_log_size":87726,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23264240,"kb_used_data":18264984,"kb_used_omap":5394,"kb_used_meta":4993837,"kb_avail":259851280,"statfs":{"total":289910292480,"available":266087710720,"internally_reserved":0,"allocated":18703343616,"data_stored":38304688981,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5523905,"internal_metadata":5113689663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,14,10,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26324992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6367,"num_read_kb":25396,"num_write":6418,"num_write_kb":25672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002024"},"pg_stats":[{"pgid":"2.7","version":"148'7649","reported_seq":11204,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:35.382171+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:22:35.382171+0000","last_peered":"2023-10-21T05:22:35.382171+0000","last_clean":"2023-10-21T05:22:35.382171+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:22:35.382171+0000","last_undegraded":"2023-10-21T05:22:35.382171+0000","last_fullsized":"2023-10-21T05:22:35.382171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"148'3914","reported_seq":7734,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:35.382887+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:22:35.382887+0000","last_peered":"2023-10-21T05:22:35.382887+0000","last_clean":"2023-10-21T05:22:35.382887+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:22:35.382887+0000","last_undegraded":"2023-10-21T05:22:35.382887+0000","last_fullsized":"2023-10-21T05:22:35.382887+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"148'11416","reported_seq":16608,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:35.387823+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:22:35.387823+0000","last_peered":"2023-10-21T05:22:35.387823+0000","last_clean":"2023-10-21T05:22:35.387823+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:22:35.387823+0000","last_undegraded":"2023-10-21T05:22:35.387823+0000","last_fullsized":"2023-10-21T05:22:35.387823+0000","mapping_epoch":11,"log_start":"55'1400","ondisk_log_start":"55'1400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"148'7690","reported_seq":12260,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:35.381929+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:22:35.381929+0000","last_peered":"2023-10-21T05:22:35.381929+0000","last_clean":"2023-10-21T05:22:35.381929+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:22:35.381929+0000","last_undegraded":"2023-10-21T05:22:35.381929+0000","last_fullsized":"2023-10-21T05:22:35.381929+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'575009","reported_seq":1136689,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:36.330179+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:22:36.330179+0000","last_peered":"2023-10-21T05:22:36.330179+0000","last_clean":"2023-10-21T05:22:36.330179+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:22:36.330179+0000","last_undegraded":"2023-10-21T05:22:36.330179+0000","last_fullsized":"2023-10-21T05:22:36.330179+0000","mapping_epoch":14,"log_start":"147'564962","ondisk_log_start":"147'564962","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7497383936,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":565984,"num_read_kb":2235732,"num_write":564854,"num_write_kb":2259416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'15514","reported_seq":22576,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:35.914134+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:22:35.914134+0000","last_peered":"2023-10-21T05:22:35.914134+0000","last_clean":"2023-10-21T05:22:35.914134+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:22:35.914134+0000","last_undegraded":"2023-10-21T05:22:35.914134+0000","last_fullsized":"2023-10-21T05:22:35.914134+0000","mapping_epoch":11,"log_start":"93'5500","ondisk_log_start":"93'5500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"148'581644","reported_seq":1151052,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:37.048848+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:22:37.048848+0000","last_peered":"2023-10-21T05:22:37.048848+0000","last_clean":"2023-10-21T05:22:37.048848+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:22:37.048848+0000","last_undegraded":"2023-10-21T05:22:37.048848+0000","last_fullsized":"2023-10-21T05:22:37.048848+0000","mapping_epoch":14,"log_start":"147'571561","ondisk_log_start":"147'571561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7589965824,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":573750,"num_read_kb":2265896,"num_write":571403,"num_write_kb":2285612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3690,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:34.498910+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:22:34.498910+0000","last_peered":"2023-10-21T05:22:34.498910+0000","last_clean":"2023-10-21T05:22:34.498910+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:22:34.498910+0000","last_undegraded":"2023-10-21T05:22:34.498910+0000","last_fullsized":"2023-10-21T05:22:34.498910+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'607824","reported_seq":1199423,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:36.352834+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:22:36.352834+0000","last_peered":"2023-10-21T05:22:36.352834+0000","last_clean":"2023-10-21T05:22:36.352834+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:22:36.352834+0000","last_undegraded":"2023-10-21T05:22:36.352834+0000","last_fullsized":"2023-10-21T05:22:36.352834+0000","mapping_epoch":14,"log_start":"147'597754","ondisk_log_start":"147'597754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7927885824,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":596122,"num_read_kb":2355148,"num_write":597110,"num_write_kb":2388440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7338,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:34.497484+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:22:34.497484+0000","last_peered":"2023-10-21T05:22:34.497484+0000","last_clean":"2023-10-21T05:22:34.497484+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:22:34.497484+0000","last_undegraded":"2023-10-21T05:22:34.497484+0000","last_fullsized":"2023-10-21T05:22:34.497484+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"148'556071","reported_seq":1103010,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:37.077417+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:22:37.077417+0000","last_peered":"2023-10-21T05:22:37.077417+0000","last_clean":"2023-10-21T05:22:37.077417+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:22:37.077417+0000","last_undegraded":"2023-10-21T05:22:37.077417+0000","last_fullsized":"2023-10-21T05:22:37.077417+0000","mapping_epoch":14,"log_start":"147'546058","ondisk_log_start":"147'546058","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2278,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 285s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7300071443,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":551173,"num_read_kb":2177612,"num_write":549345,"num_write_kb":2197374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"148'7608","reported_seq":11095,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:35.379205+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:22:35.379205+0000","last_peered":"2023-10-21T05:22:35.379205+0000","last_clean":"2023-10-21T05:22:35.379205+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:22:35.379205+0000","last_undegraded":"2023-10-21T05:22:35.379205+0000","last_fullsized":"2023-10-21T05:22:35.379205+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":327,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:21:49.546764+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:21:49.546764+0000","last_peered":"2023-10-21T05:21:49.546764+0000","last_clean":"2023-10-21T05:21:49.546764+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:21:49.546764+0000","last_undegraded":"2023-10-21T05:21:49.546764+0000","last_fullsized":"2023-10-21T05:21:49.546764+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30315307027,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2287029,"num_read_kb":9034388,"num_write":2282712,"num_write_kb":9130842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18699845632,"data_stored":38303662118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47414,"ondisk_log_size":47414,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739366,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11031096,"kb_used_data":9131248,"kb_used_omap":2691,"kb_used_meta":1897148,"kb_avail":83340744,"statfs":{"total":96636764160,"available":85340921856,"internally_reserved":0,"allocated":9350397952,"data_stored":19151286514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":1942679697},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.94799999999999995,"15min":1.034},"min":{"1min":0.215,"5min":0.186,"15min":0.186},"max":{"1min":2.7909999999999999,"5min":4.5659999999999998,"15min":27.184999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.92500000000000004,"15min":0.99199999999999999},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.819,"5min":6.306,"15min":27.251000000000001},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.93799999999999994,"15min":0.96299999999999997},"min":{"1min":0.29699999999999999,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":2.286,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.0720000000000001,"15min":1.0880000000000001},"min":{"1min":0.29099999999999998,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":10.253},"last":0.94999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739365,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6354756,"kb_used_data":4484604,"kb_used_omap":1192,"kb_used_meta":1868951,"kb_avail":88017084,"statfs":{"total":96636764160,"available":90129494016,"internally_reserved":0,"allocated":4592234496,"data_stored":9403289798,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1913806771},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,9,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.74099999999999999,"15min":0.77100000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.16500000000000001},"max":{"1min":3.8879999999999999,"5min":4.1050000000000004,"15min":6.2140000000000004},"last":0.185},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79800000000000004,"15min":0.79600000000000004},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.20899999999999999},"max":{"1min":3.8530000000000002,"5min":4.1879999999999997,"15min":6.0579999999999998},"last":0.38}]},{"osd":2,"last update":"Sat Oct 21 05:22:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.74399999999999999,"15min":0.89200000000000002},"min":{"1min":0.30499999999999999,"5min":0.26200000000000001,"15min":0.19900000000000001},"max":{"1min":1.9670000000000001,"5min":4.1459999999999999,"15min":10.167999999999999},"last":0.222},{"interface":"front","average":{"1min":1.014,"5min":0.79600000000000004,"15min":0.89000000000000001},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.202,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.64400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5878388,"kb_used_data":4649132,"kb_used_omap":1510,"kb_used_meta":1227737,"kb_avail":88493452,"statfs":{"total":96636764160,"available":90617294848,"internally_reserved":0,"allocated":4760711168,"data_stored":9750112669,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1257203195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71499999999999997,"15min":0.80300000000000005},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.187},"max":{"1min":2.5579999999999998,"5min":4.5570000000000004,"15min":5.0339999999999998},"last":5.923},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68799999999999994,"15min":0.873},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20899999999999999},"max":{"1min":2.524,"5min":5.968,"15min":6.4050000000000002},"last":5.8879999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:22:27 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.89200000000000002,"15min":0.91400000000000003},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":2.6019999999999999,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":14.053000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.036,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":4.0129999999999999,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":2.4809999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4759425024,"data_stored":9749471232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4590592000,"data_stored":9402454035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9349828608,"data_stored":19151736851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:38.902 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:38.903 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:42.818 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:22:42.810+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:22:49.834 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:22:49.822+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:22:49.834 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:22:49.822+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:22:54.751 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.745+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:22:54.755 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.745+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:22:54.759 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.757+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:22:54.759 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.757+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:22:54.759 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.757+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:22:54.765 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.762+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:22:54.765 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.762+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:22:54.766 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.762+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:22:54.766 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.762+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:22:54.766 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:22:54.762+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:22:58.904 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:59.451 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:22:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:22:59.472 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2646,"stamp":"2023-10-21T05:22:57.946976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30351279869,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2344626,"num_read_kb":9107930,"num_write":2292474,"num_write_kb":9167671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87857,"ondisk_log_size":87857,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23682184,"kb_used_data":18332896,"kb_used_omap":5394,"kb_used_meta":5343853,"kb_avail":259433336,"statfs":{"total":289910292480,"available":265659736064,"internally_reserved":0,"allocated":18772885504,"data_stored":38374219343,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5523905,"internal_metadata":5472106047},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,2,5,11,8,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17498112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4021,"num_read_kb":16044,"num_write":4256,"num_write_kb":17024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001662"},"pg_stats":[{"pgid":"2.7","version":"149'7681","reported_seq":11246,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.180253+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:22:56.180253+0000","last_peered":"2023-10-21T05:22:56.180253+0000","last_clean":"2023-10-21T05:22:56.180253+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:22:56.180253+0000","last_undegraded":"2023-10-21T05:22:56.180253+0000","last_fullsized":"2023-10-21T05:22:56.180253+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7681,"log_dups_size":0,"ondisk_log_size":7681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'3930","reported_seq":7755,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:54.977645+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:22:54.977645+0000","last_peered":"2023-10-21T05:22:54.977645+0000","last_clean":"2023-10-21T05:22:54.977645+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:22:54.977645+0000","last_undegraded":"2023-10-21T05:22:54.977645+0000","last_fullsized":"2023-10-21T05:22:54.977645+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'11464","reported_seq":16669,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.180965+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:22:56.180965+0000","last_peered":"2023-10-21T05:22:56.180965+0000","last_clean":"2023-10-21T05:22:56.180965+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:22:56.180965+0000","last_undegraded":"2023-10-21T05:22:56.180965+0000","last_fullsized":"2023-10-21T05:22:56.180965+0000","mapping_epoch":11,"log_start":"55'1400","ondisk_log_start":"55'1400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'7722","reported_seq":12301,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.168847+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:22:56.168847+0000","last_peered":"2023-10-21T05:22:56.168847+0000","last_clean":"2023-10-21T05:22:56.168847+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:22:56.168847+0000","last_undegraded":"2023-10-21T05:22:56.168847+0000","last_fullsized":"2023-10-21T05:22:56.168847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'577099","reported_seq":1140816,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.588984+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:22:56.588984+0000","last_peered":"2023-10-21T05:22:56.588984+0000","last_clean":"2023-10-21T05:22:56.588984+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:22:56.588984+0000","last_undegraded":"2023-10-21T05:22:56.588984+0000","last_fullsized":"2023-10-21T05:22:56.588984+0000","mapping_epoch":14,"log_start":"147'567062","ondisk_log_start":"147'567062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7505997824,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":568019,"num_read_kb":2243848,"num_write":566944,"num_write_kb":2267776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'15618","reported_seq":22697,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:54.979220+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:22:54.979220+0000","last_peered":"2023-10-21T05:22:54.979220+0000","last_clean":"2023-10-21T05:22:54.979220+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:22:54.979220+0000","last_undegraded":"2023-10-21T05:22:54.979220+0000","last_fullsized":"2023-10-21T05:22:54.979220+0000","mapping_epoch":11,"log_start":"95'5600","ondisk_log_start":"95'5600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'583783","reported_seq":1155195,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:57.192397+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:22:57.192397+0000","last_peered":"2023-10-21T05:22:57.192397+0000","last_clean":"2023-10-21T05:22:57.192397+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:22:57.192397+0000","last_undegraded":"2023-10-21T05:22:57.192397+0000","last_fullsized":"2023-10-21T05:22:57.192397+0000","mapping_epoch":14,"log_start":"147'573761","ondisk_log_start":"147'573761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7598718976,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":575751,"num_read_kb":2273888,"num_write":573542,"num_write_kb":2294168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3699,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:54.978040+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:22:54.978040+0000","last_peered":"2023-10-21T05:22:54.978040+0000","last_clean":"2023-10-21T05:22:54.978040+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:22:54.978040+0000","last_undegraded":"2023-10-21T05:22:54.978040+0000","last_fullsized":"2023-10-21T05:22:54.978040+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'610101","reported_seq":1203865,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.569334+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:22:56.569334+0000","last_peered":"2023-10-21T05:22:56.569334+0000","last_clean":"2023-10-21T05:22:56.569334+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:22:56.569334+0000","last_undegraded":"2023-10-21T05:22:56.569334+0000","last_fullsized":"2023-10-21T05:22:56.569334+0000","mapping_epoch":14,"log_start":"147'600054","ondisk_log_start":"147'600054","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7937269760,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":598287,"num_read_kb":2363780,"num_write":599387,"num_write_kb":2397548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7355,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:54.978490+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:22:54.978490+0000","last_peered":"2023-10-21T05:22:54.978490+0000","last_clean":"2023-10-21T05:22:54.978490+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:22:54.978490+0000","last_undegraded":"2023-10-21T05:22:54.978490+0000","last_fullsized":"2023-10-21T05:22:54.978490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'558232","reported_seq":1107114,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:57.196098+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:22:57.196098+0000","last_peered":"2023-10-21T05:22:57.196098+0000","last_clean":"2023-10-21T05:22:57.196098+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:22:57.196098+0000","last_undegraded":"2023-10-21T05:22:57.196098+0000","last_fullsized":"2023-10-21T05:22:57.196098+0000","mapping_epoch":14,"log_start":"147'548158","ondisk_log_start":"147'548158","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2356,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 305s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7308697619,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":553110,"num_read_kb":2185344,"num_write":551443,"num_write_kb":2205766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'7640","reported_seq":11136,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.174326+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:22:56.174326+0000","last_peered":"2023-10-21T05:22:56.174326+0000","last_clean":"2023-10-21T05:22:56.174326+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:22:56.174326+0000","last_undegraded":"2023-10-21T05:22:56.174326+0000","last_fullsized":"2023-10-21T05:22:56.174326+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:27.012284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:22:27.012284+0000","last_peered":"2023-10-21T05:22:27.012284+0000","last_clean":"2023-10-21T05:22:27.012284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:22:27.012284+0000","last_undegraded":"2023-10-21T05:22:27.012284+0000","last_fullsized":"2023-10-21T05:22:27.012284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30350684179,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2295167,"num_read_kb":9066860,"num_write":2291316,"num_write_kb":9165258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18770153472,"data_stored":38373969958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40180,"ondisk_log_size":40180,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47578,"ondisk_log_size":47578,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739370,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11252692,"kb_used_data":9165188,"kb_used_omap":2691,"kb_used_meta":2084796,"kb_avail":83119148,"statfs":{"total":96636764160,"available":85114007552,"internally_reserved":0,"allocated":9385152512,"data_stored":19186037232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2756463,"internal_metadata":2134831249},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.94799999999999995,"15min":1.034},"min":{"1min":0.215,"5min":0.186,"15min":0.186},"max":{"1min":2.7909999999999999,"5min":4.5659999999999998,"15min":27.184999999999999},"last":2.0579999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.92500000000000004,"15min":0.99199999999999999},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.819,"5min":6.306,"15min":27.251000000000001},"last":2.1320000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.93799999999999994,"15min":0.96299999999999997},"min":{"1min":0.29699999999999999,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":2.286,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":2.0899999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.0720000000000001,"15min":1.0880000000000001},"min":{"1min":0.29099999999999998,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":10.253},"last":0.95299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739369,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6443672,"kb_used_data":4501320,"kb_used_omap":1192,"kb_used_meta":1941143,"kb_avail":87928168,"statfs":{"total":96636764160,"available":90038444032,"internally_reserved":0,"allocated":4609351680,"data_stored":9420403140,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":1987731379},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.74099999999999999,"15min":0.77100000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.16500000000000001},"max":{"1min":3.8879999999999999,"5min":4.1050000000000004,"15min":6.2140000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79800000000000004,"15min":0.79600000000000004},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.20899999999999999},"max":{"1min":3.8530000000000002,"5min":4.1879999999999997,"15min":6.0579999999999998},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:22:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.74399999999999999,"15min":0.89200000000000002},"min":{"1min":0.30499999999999999,"5min":0.26200000000000001,"15min":0.19900000000000001},"max":{"1min":1.9670000000000001,"5min":4.1459999999999999,"15min":10.167999999999999},"last":2.8860000000000001},{"interface":"front","average":{"1min":1.014,"5min":0.79600000000000004,"15min":0.89000000000000001},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.202,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":2.9129999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5985820,"kb_used_data":4666388,"kb_used_omap":1510,"kb_used_meta":1317913,"kb_avail":88386020,"statfs":{"total":96636764160,"available":90507284480,"internally_reserved":0,"allocated":4778381312,"data_stored":9767778971,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1349543419},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71499999999999997,"15min":0.80300000000000005},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.187},"max":{"1min":2.5579999999999998,"5min":4.5570000000000004,"15min":5.0339999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68799999999999994,"15min":0.873},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20899999999999999},"max":{"1min":2.524,"5min":5.968,"15min":6.4050000000000002},"last":0.627}]},{"osd":2,"last update":"Sat Oct 21 05:22:27 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.89200000000000002,"15min":0.91400000000000003},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":2.6019999999999999,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":1.083},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.036,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":4.0129999999999999,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4777295872,"data_stored":9767342080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4607934464,"data_stored":9419796499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9384923136,"data_stored":19186831379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:59.474 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:59.475 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:16.738 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:16.737+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:23:16.739 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:16.737+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:23:16.739 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:16.737+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:23:16.740 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:16.737+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:23:19.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:19.738 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:19.735+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:23:19.738 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:19.735+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:23:19.738 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:19.735+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:23:19.739 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:19.735+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:23:19.739 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:19.735+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:23:20.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:23:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:23:20.056 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2657,"stamp":"2023-10-21T05:23:19.950696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30386669309,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2353266,"num_read_kb":9142414,"num_write":2301089,"num_write_kb":9202131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87978,"ondisk_log_size":87978,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23515272,"kb_used_data":18402860,"kb_used_omap":5488,"kb_used_meta":5106895,"kb_avail":259600248,"statfs":{"total":289910292480,"available":265830653952,"internally_reserved":0,"allocated":18844528640,"data_stored":38445862479,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5619949,"internal_metadata":5229461267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,26,2,5,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17879040,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4347,"num_read_kb":17360,"num_write":4355,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002012"},"pg_stats":[{"pgid":"2.7","version":"149'7699","reported_seq":11272,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:16.578285+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:23:16.578285+0000","last_peered":"2023-10-21T05:23:16.578285+0000","last_clean":"2023-10-21T05:23:16.578285+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:23:16.578285+0000","last_undegraded":"2023-10-21T05:23:16.578285+0000","last_fullsized":"2023-10-21T05:23:16.578285+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7699,"log_dups_size":0,"ondisk_log_size":7699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'3939","reported_seq":7768,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.457623+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:23:15.457623+0000","last_peered":"2023-10-21T05:23:15.457623+0000","last_clean":"2023-10-21T05:23:15.457623+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:23:15.457623+0000","last_undegraded":"2023-10-21T05:23:15.457623+0000","last_fullsized":"2023-10-21T05:23:15.457623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'11491","reported_seq":16708,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:16.589260+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:23:16.589260+0000","last_peered":"2023-10-21T05:23:16.589260+0000","last_clean":"2023-10-21T05:23:16.589260+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:23:16.589260+0000","last_undegraded":"2023-10-21T05:23:16.589260+0000","last_fullsized":"2023-10-21T05:23:16.589260+0000","mapping_epoch":11,"log_start":"55'1400","ondisk_log_start":"55'1400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'7740","reported_seq":12327,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:16.569413+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:23:16.569413+0000","last_peered":"2023-10-21T05:23:16.569413+0000","last_clean":"2023-10-21T05:23:16.569413+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:23:16.569413+0000","last_undegraded":"2023-10-21T05:23:16.569413+0000","last_fullsized":"2023-10-21T05:23:16.569413+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'579196","reported_seq":1145036,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:16.715096+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:23:16.715096+0000","last_peered":"2023-10-21T05:23:16.715096+0000","last_clean":"2023-10-21T05:23:16.715096+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:23:16.715096+0000","last_undegraded":"2023-10-21T05:23:16.715096+0000","last_fullsized":"2023-10-21T05:23:16.715096+0000","mapping_epoch":14,"log_start":"147'569162","ondisk_log_start":"147'569162","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7514619904,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":570143,"num_read_kb":2252316,"num_write":569041,"num_write_kb":2276164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"149'15658","reported_seq":22753,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.457122+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:23:15.457122+0000","last_peered":"2023-10-21T05:23:15.457122+0000","last_clean":"2023-10-21T05:23:15.457122+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:23:15.457122+0000","last_undegraded":"2023-10-21T05:23:15.457122+0000","last_fullsized":"2023-10-21T05:23:15.457122+0000","mapping_epoch":11,"log_start":"95'5600","ondisk_log_start":"95'5600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'585994","reported_seq":1159618,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:17.486380+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:23:17.486380+0000","last_peered":"2023-10-21T05:23:17.486380+0000","last_clean":"2023-10-21T05:23:17.486380+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:23:17.486380+0000","last_undegraded":"2023-10-21T05:23:17.486380+0000","last_fullsized":"2023-10-21T05:23:17.486380+0000","mapping_epoch":14,"log_start":"147'575961","ondisk_log_start":"147'575961","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7607820288,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":577963,"num_read_kb":2282732,"num_write":575753,"num_write_kb":2303012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3707,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.464094+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:23:15.464094+0000","last_peered":"2023-10-21T05:23:15.464094+0000","last_clean":"2023-10-21T05:23:15.464094+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:23:15.464094+0000","last_undegraded":"2023-10-21T05:23:15.464094+0000","last_fullsized":"2023-10-21T05:23:15.464094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'612342","reported_seq":1208339,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:16.707315+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:23:16.707315+0000","last_peered":"2023-10-21T05:23:16.707315+0000","last_clean":"2023-10-21T05:23:16.707315+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:23:16.707315+0000","last_undegraded":"2023-10-21T05:23:16.707315+0000","last_fullsized":"2023-10-21T05:23:16.707315+0000","mapping_epoch":14,"log_start":"147'602254","ondisk_log_start":"147'602254","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7946469376,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":600520,"num_read_kb":2372684,"num_write":601628,"num_write_kb":2406512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7371,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:15.470725+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:23:15.470725+0000","last_peered":"2023-10-21T05:23:15.470725+0000","last_clean":"2023-10-21T05:23:15.470725+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:23:15.470725+0000","last_undegraded":"2023-10-21T05:23:15.470725+0000","last_fullsized":"2023-10-21T05:23:15.470725+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'560374","reported_seq":1111340,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:17.511710+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:23:17.511710+0000","last_peered":"2023-10-21T05:23:17.511710+0000","last_clean":"2023-10-21T05:23:17.511710+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:23:17.511710+0000","last_undegraded":"2023-10-21T05:23:17.511710+0000","last_fullsized":"2023-10-21T05:23:17.511710+0000","mapping_epoch":14,"log_start":"147'550358","ondisk_log_start":"147'550358","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2449,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 326s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7317164051,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":555181,"num_read_kb":2193612,"num_write":553509,"num_write_kb":2214030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'7658","reported_seq":11162,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:16.584045+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:23:16.584045+0000","last_peered":"2023-10-21T05:23:16.584045+0000","last_clean":"2023-10-21T05:23:16.584045+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:23:16.584045+0000","last_undegraded":"2023-10-21T05:23:16.584045+0000","last_fullsized":"2023-10-21T05:23:16.584045+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:27.012284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:22:27.012284+0000","last_peered":"2023-10-21T05:22:27.012284+0000","last_clean":"2023-10-21T05:22:27.012284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:22:27.012284+0000","last_undegraded":"2023-10-21T05:22:27.012284+0000","last_fullsized":"2023-10-21T05:22:27.012284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30386073619,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2303807,"num_read_kb":9101344,"num_write":2299931,"num_write_kb":9199718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18840559616,"data_stored":38444376102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40171,"ondisk_log_size":40171,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47708,"ondisk_log_size":47708,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739374,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10855112,"kb_used_data":9200764,"kb_used_omap":2785,"kb_used_meta":1651550,"kb_avail":83516728,"statfs":{"total":96636764160,"available":85521129472,"internally_reserved":0,"allocated":9421582336,"data_stored":19222467056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2852507,"internal_metadata":1691187557},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.94799999999999995,"15min":1.034},"min":{"1min":0.215,"5min":0.186,"15min":0.186},"max":{"1min":2.7909999999999999,"5min":4.5659999999999998,"15min":27.184999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.92500000000000004,"15min":0.99199999999999999},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.819,"5min":6.306,"15min":27.251000000000001},"last":0.32500000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.93799999999999994,"15min":0.96299999999999997},"min":{"1min":0.29699999999999999,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":2.286,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.0720000000000001,"15min":1.0880000000000001},"min":{"1min":0.29099999999999998,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":10.253},"last":0.74299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739373,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6558760,"kb_used_data":4518368,"kb_used_omap":1192,"kb_used_meta":2039191,"kb_avail":87813080,"statfs":{"total":96636764160,"available":89920593920,"internally_reserved":0,"allocated":4626808832,"data_stored":9437860292,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1220685,"internal_metadata":2088132531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.74099999999999999,"15min":0.77100000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.16500000000000001},"max":{"1min":3.8879999999999999,"5min":4.1050000000000004,"15min":6.2140000000000004},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79800000000000004,"15min":0.79600000000000004},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.20899999999999999},"max":{"1min":3.8530000000000002,"5min":4.1879999999999997,"15min":6.0579999999999998},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.74399999999999999,"15min":0.89200000000000002},"min":{"1min":0.30499999999999999,"5min":0.26200000000000001,"15min":0.19900000000000001},"max":{"1min":1.9670000000000001,"5min":4.1459999999999999,"15min":10.167999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.014,"5min":0.79600000000000004,"15min":0.89000000000000001},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.202,"5min":5.9409999999999998,"15min":6.1230000000000002},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6101400,"kb_used_data":4683728,"kb_used_omap":1510,"kb_used_meta":1416153,"kb_avail":88270440,"statfs":{"total":96636764160,"available":90388930560,"internally_reserved":0,"allocated":4796137472,"data_stored":9785535131,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1450141179},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,23,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71499999999999997,"15min":0.80300000000000005},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.187},"max":{"1min":2.5579999999999998,"5min":4.5570000000000004,"15min":5.0339999999999998},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68799999999999994,"15min":0.873},"min":{"1min":0.30199999999999999,"5min":0.223,"15min":0.20899999999999999},"max":{"1min":2.524,"5min":5.968,"15min":6.4050000000000002},"last":1.1479999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:22:27 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.89200000000000002,"15min":0.91400000000000003},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":2.6019999999999999,"5min":6.1029999999999998,"15min":6.1029999999999998},"last":4.1870000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.036,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":4.0129999999999999,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4795072512,"data_stored":9785118720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4625473536,"data_stored":9437335571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9420013568,"data_stored":19221921811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:20.058 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:20.058 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:23.694 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:23.691+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:23:26.658 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:26.646+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:23:38.600 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:38.598+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:23:40.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:40.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:23:40.568 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:23:40.593 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2667,"stamp":"2023-10-21T05:23:39.954831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30422214397,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2361824,"num_read_kb":9176562,"num_write":2309729,"num_write_kb":9236691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88014,"ondisk_log_size":88014,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23255920,"kb_used_data":18471640,"kb_used_omap":5634,"kb_used_meta":4778621,"kb_avail":259859600,"statfs":{"total":289910292480,"available":266096230400,"internally_reserved":0,"allocated":18914959360,"data_stored":38516293199,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5769397,"internal_metadata":4893308747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,8,7,10,14,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17649664,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4292,"num_read_kb":17128,"num_write":4295,"num_write_kb":17180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002068"},"pg_stats":[{"pgid":"2.7","version":"149'7725","reported_seq":11306,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.937747+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:23:35.937747+0000","last_peered":"2023-10-21T05:23:35.937747+0000","last_clean":"2023-10-21T05:23:35.937747+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:23:35.937747+0000","last_undegraded":"2023-10-21T05:23:35.937747+0000","last_fullsized":"2023-10-21T05:23:35.937747+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7725,"log_dups_size":0,"ondisk_log_size":7725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'3952","reported_seq":7786,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:33.947244+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:23:33.947244+0000","last_peered":"2023-10-21T05:23:33.947244+0000","last_clean":"2023-10-21T05:23:33.947244+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:23:33.947244+0000","last_undegraded":"2023-10-21T05:23:33.947244+0000","last_fullsized":"2023-10-21T05:23:33.947244+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3952,"log_dups_size":0,"ondisk_log_size":3952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'11530","reported_seq":16759,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.937870+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:23:35.937870+0000","last_peered":"2023-10-21T05:23:35.937870+0000","last_clean":"2023-10-21T05:23:35.937870+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:23:35.937870+0000","last_undegraded":"2023-10-21T05:23:35.937870+0000","last_fullsized":"2023-10-21T05:23:35.937870+0000","mapping_epoch":11,"log_start":"57'1500","ondisk_log_start":"57'1500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'7766","reported_seq":12361,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.938025+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:23:35.938025+0000","last_peered":"2023-10-21T05:23:35.938025+0000","last_clean":"2023-10-21T05:23:35.938025+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:23:35.938025+0000","last_undegraded":"2023-10-21T05:23:35.938025+0000","last_fullsized":"2023-10-21T05:23:35.938025+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7766,"log_dups_size":0,"ondisk_log_size":7766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'581278","reported_seq":1149308,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:36.829816+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:23:36.829816+0000","last_peered":"2023-10-21T05:23:36.829816+0000","last_clean":"2023-10-21T05:23:36.829816+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:23:36.829816+0000","last_undegraded":"2023-10-21T05:23:36.829816+0000","last_fullsized":"2023-10-21T05:23:36.829816+0000","mapping_epoch":14,"log_start":"148'571262","ondisk_log_start":"148'571262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7523176448,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":572332,"num_read_kb":2261056,"num_write":571123,"num_write_kb":2284492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"149'15746","reported_seq":22857,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:36.592190+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:23:36.592190+0000","last_peered":"2023-10-21T05:23:36.592190+0000","last_clean":"2023-10-21T05:23:36.592190+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:23:36.592190+0000","last_undegraded":"2023-10-21T05:23:36.592190+0000","last_fullsized":"2023-10-21T05:23:36.592190+0000","mapping_epoch":11,"log_start":"95'5700","ondisk_log_start":"95'5700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'588218","reported_seq":1163948,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:37.682621+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:23:37.682621+0000","last_peered":"2023-10-21T05:23:37.682621+0000","last_clean":"2023-10-21T05:23:37.682621+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:23:37.682621+0000","last_undegraded":"2023-10-21T05:23:37.682621+0000","last_fullsized":"2023-10-21T05:23:37.682621+0000","mapping_epoch":14,"log_start":"148'578161","ondisk_log_start":"148'578161","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7616921600,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":580070,"num_read_kb":2291124,"num_write":577977,"num_write_kb":2311908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3716,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.964236+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:23:35.964236+0000","last_peered":"2023-10-21T05:23:35.964236+0000","last_clean":"2023-10-21T05:23:35.964236+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:23:35.964236+0000","last_undegraded":"2023-10-21T05:23:35.964236+0000","last_fullsized":"2023-10-21T05:23:35.964236+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'614587","reported_seq":1212792,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:36.829336+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:23:36.829336+0000","last_peered":"2023-10-21T05:23:36.829336+0000","last_clean":"2023-10-21T05:23:36.829336+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:23:36.829336+0000","last_undegraded":"2023-10-21T05:23:36.829336+0000","last_fullsized":"2023-10-21T05:23:36.829336+0000","mapping_epoch":14,"log_start":"148'604554","ondisk_log_start":"148'604554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7955746816,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":602726,"num_read_kb":2381492,"num_write":603873,"num_write_kb":2415492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7387,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.941184+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:23:35.941184+0000","last_peered":"2023-10-21T05:23:35.941184+0000","last_clean":"2023-10-21T05:23:35.941184+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:23:35.941184+0000","last_undegraded":"2023-10-21T05:23:35.941184+0000","last_fullsized":"2023-10-21T05:23:35.941184+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'562541","reported_seq":1115569,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:37.682652+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:23:37.682652+0000","last_peered":"2023-10-21T05:23:37.682652+0000","last_clean":"2023-10-21T05:23:37.682652+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:23:37.682652+0000","last_undegraded":"2023-10-21T05:23:37.682652+0000","last_fullsized":"2023-10-21T05:23:37.682652+0000","mapping_epoch":14,"log_start":"148'552458","ondisk_log_start":"148'552458","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2544,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 346s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7325773843,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":557237,"num_read_kb":2201820,"num_write":555598,"num_write_kb":2222386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'7684","reported_seq":11196,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:35.937855+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:23:35.937855+0000","last_peered":"2023-10-21T05:23:35.937855+0000","last_clean":"2023-10-21T05:23:35.937855+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:23:35.937855+0000","last_undegraded":"2023-10-21T05:23:35.937855+0000","last_fullsized":"2023-10-21T05:23:35.937855+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:27.012284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:22:27.012284+0000","last_peered":"2023-10-21T05:22:27.012284+0000","last_clean":"2023-10-21T05:22:27.012284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:22:27.012284+0000","last_undegraded":"2023-10-21T05:22:27.012284+0000","last_fullsized":"2023-10-21T05:22:27.012284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30421618707,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2312365,"num_read_kb":9135492,"num_write":2308571,"num_write_kb":9234278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18911051776,"data_stored":38514868262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40189,"ondisk_log_size":40189,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47726,"ondisk_log_size":47726,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739378,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11042100,"kb_used_data":9235052,"kb_used_omap":2785,"kb_used_meta":1804254,"kb_avail":83329740,"statfs":{"total":96636764160,"available":85329653760,"internally_reserved":0,"allocated":9456693248,"data_stored":19257577968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2852507,"internal_metadata":1847556453},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.89400000000000002,"15min":1.0089999999999999},"min":{"1min":0.26400000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":2.0579999999999998,"5min":4.5659999999999998,"15min":27.184999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85899999999999999,"15min":0.97899999999999998},"min":{"1min":0.254,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.1320000000000001,"5min":5.0620000000000003,"15min":27.251000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.90900000000000003,"15min":0.92700000000000005},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.0899999999999999,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.056,"15min":1.0629999999999999},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.855,"5min":5.7430000000000003,"15min":10.253},"last":1.546}]}]},{"osd":1,"up_from":8,"seq":34359739377,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5984864,"kb_used_data":4535512,"kb_used_omap":1338,"kb_used_meta":1448005,"kb_avail":88386976,"statfs":{"total":96636764160,"available":90508263424,"internally_reserved":0,"allocated":4644364288,"data_stored":9455415748,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1482758123},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,5,6,13,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.71599999999999997,"15min":0.78100000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":4.5010000000000003,"5min":4.5010000000000003,"15min":6.2140000000000004},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77500000000000002,"15min":0.80400000000000005},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.20899999999999999},"max":{"1min":4.5750000000000002,"5min":4.5750000000000002,"15min":6.0579999999999998},"last":0.28599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:23:20 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.77900000000000003,"15min":0.88100000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":4.54,"5min":4.54,"15min":5.0339999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.044,"5min":0.81200000000000006,"15min":0.88900000000000001},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":5.1989999999999998,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":0.82599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6228956,"kb_used_data":4701076,"kb_used_omap":1510,"kb_used_meta":1526361,"kb_avail":88142884,"statfs":{"total":96636764160,"available":90258313216,"internally_reserved":0,"allocated":4813901824,"data_stored":9803299483,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1562994171},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.74199999999999999,"15min":0.81299999999999994},"min":{"1min":0.23300000000000001,"5min":0.189,"15min":0.187},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.76200000000000001,"15min":0.88200000000000001},"min":{"1min":0.253,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":5.8879999999999999,"5min":5.968,"15min":6.4050000000000002},"last":0.438}]},{"osd":2,"last update":"Sat Oct 21 05:23:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":1.1379999999999999,"15min":0.99099999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.20100000000000001},"max":{"1min":14.053000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.21,"5min":1.101,"15min":0.96299999999999997},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.831,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.81999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4812787712,"data_stored":9802833920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4643119104,"data_stored":9454981139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9455144960,"data_stored":19257053203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:40.595 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:40.596 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:47.812 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:23:47.813 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:23:47.813 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:23:47.813 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:23:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:23:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:23:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:23:47.814 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:23:47.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:23:47.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:23:47.809+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:23:58.876 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:58.871+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:23:58.877 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:58.871+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:23:58.877 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:58.871+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:23:58.877 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:23:58.871+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:24:00.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:24:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:24:01.154 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2677,"stamp":"2023-10-21T05:23:59.958683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30456207101,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2369911,"num_read_kb":9208858,"num_write":2318007,"num_write_kb":9269803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88154,"ondisk_log_size":88154,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23654676,"kb_used_data":18538232,"kb_used_omap":5634,"kb_used_meta":5110781,"kb_avail":259460844,"statfs":{"total":289910292480,"available":265687904256,"internally_reserved":0,"allocated":18983149568,"data_stored":38584483407,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5769397,"internal_metadata":5233440587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,2,13,6,22,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16924672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4084,"num_read_kb":16316,"num_write":4121,"num_write_kb":16484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002510"},"pg_stats":[{"pgid":"2.7","version":"149'7761","reported_seq":11350,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:57.679425+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:23:57.679425+0000","last_peered":"2023-10-21T05:23:57.679425+0000","last_clean":"2023-10-21T05:23:57.679425+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:23:57.679425+0000","last_undegraded":"2023-10-21T05:23:57.679425+0000","last_fullsized":"2023-10-21T05:23:57.679425+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7761,"log_dups_size":0,"ondisk_log_size":7761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'3971","reported_seq":7809,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.790020+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:23:56.790020+0000","last_peered":"2023-10-21T05:23:56.790020+0000","last_clean":"2023-10-21T05:23:56.790020+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:23:56.790020+0000","last_undegraded":"2023-10-21T05:23:56.790020+0000","last_fullsized":"2023-10-21T05:23:56.790020+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'11584","reported_seq":16826,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:57.682609+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:23:57.682609+0000","last_peered":"2023-10-21T05:23:57.682609+0000","last_clean":"2023-10-21T05:23:57.682609+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:23:57.682609+0000","last_undegraded":"2023-10-21T05:23:57.682609+0000","last_fullsized":"2023-10-21T05:23:57.682609+0000","mapping_epoch":11,"log_start":"57'1500","ondisk_log_start":"57'1500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'7803","reported_seq":12407,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:57.679277+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:23:57.679277+0000","last_peered":"2023-10-21T05:23:57.679277+0000","last_clean":"2023-10-21T05:23:57.679277+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:23:57.679277+0000","last_undegraded":"2023-10-21T05:23:57.679277+0000","last_fullsized":"2023-10-21T05:23:57.679277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7803,"log_dups_size":0,"ondisk_log_size":7803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'583324","reported_seq":1153368,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.938738+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:23:56.938738+0000","last_peered":"2023-10-21T05:23:56.938738+0000","last_clean":"2023-10-21T05:23:56.938738+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:23:56.938738+0000","last_undegraded":"2023-10-21T05:23:56.938738+0000","last_fullsized":"2023-10-21T05:23:56.938738+0000","mapping_epoch":14,"log_start":"148'573262","ondisk_log_start":"148'573262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7531606016,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":574344,"num_read_kb":2269088,"num_write":573169,"num_write_kb":2292676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"149'15798","reported_seq":22926,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.417089+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:23:56.417089+0000","last_peered":"2023-10-21T05:23:56.417089+0000","last_clean":"2023-10-21T05:23:56.417089+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:23:56.417089+0000","last_undegraded":"2023-10-21T05:23:56.417089+0000","last_fullsized":"2023-10-21T05:23:56.417089+0000","mapping_epoch":11,"log_start":"95'5700","ondisk_log_start":"95'5700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'590281","reported_seq":1168067,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:57.740831+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:23:57.740831+0000","last_peered":"2023-10-21T05:23:57.740831+0000","last_clean":"2023-10-21T05:23:57.740831+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:23:57.740831+0000","last_undegraded":"2023-10-21T05:23:57.740831+0000","last_fullsized":"2023-10-21T05:23:57.740831+0000","mapping_epoch":14,"log_start":"148'580261","ondisk_log_start":"148'580261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7625400320,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":582124,"num_read_kb":2299328,"num_write":580040,"num_write_kb":2320160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3724,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.417665+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:23:56.417665+0000","last_peered":"2023-10-21T05:23:56.417665+0000","last_clean":"2023-10-21T05:23:56.417665+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:23:56.417665+0000","last_undegraded":"2023-10-21T05:23:56.417665+0000","last_fullsized":"2023-10-21T05:23:56.417665+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'616756","reported_seq":1217018,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.939094+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:23:56.939094+0000","last_peered":"2023-10-21T05:23:56.939094+0000","last_clean":"2023-10-21T05:23:56.939094+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:23:56.939094+0000","last_undegraded":"2023-10-21T05:23:56.939094+0000","last_fullsized":"2023-10-21T05:23:56.939094+0000","mapping_epoch":14,"log_start":"148'606754","ondisk_log_start":"148'606754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7964635136,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":604784,"num_read_kb":2389712,"num_write":606042,"num_write_kb":2424168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7403,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:56.424573+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:23:56.424573+0000","last_peered":"2023-10-21T05:23:56.424573+0000","last_clean":"2023-10-21T05:23:56.424573+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:23:56.424573+0000","last_undegraded":"2023-10-21T05:23:56.424573+0000","last_fullsized":"2023-10-21T05:23:56.424573+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'564669","reported_seq":1119676,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:57.729999+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:23:57.729999+0000","last_peered":"2023-10-21T05:23:57.729999+0000","last_clean":"2023-10-21T05:23:57.729999+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:23:57.729999+0000","last_undegraded":"2023-10-21T05:23:57.729999+0000","last_fullsized":"2023-10-21T05:23:57.729999+0000","mapping_epoch":14,"log_start":"148'554658","ondisk_log_start":"148'554658","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2701,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 366s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7333969939,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":559200,"num_read_kb":2209660,"num_write":557598,"num_write_kb":2230386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'7720","reported_seq":11241,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:23:57.682459+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:23:57.682459+0000","last_peered":"2023-10-21T05:23:57.682459+0000","last_clean":"2023-10-21T05:23:57.682459+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:23:57.682459+0000","last_undegraded":"2023-10-21T05:23:57.682459+0000","last_fullsized":"2023-10-21T05:23:57.682459+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-21T05:22:27.012284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:22:27.012284+0000","last_peered":"2023-10-21T05:22:27.012284+0000","last_clean":"2023-10-21T05:22:27.012284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:22:27.012284+0000","last_undegraded":"2023-10-21T05:22:27.012284+0000","last_fullsized":"2023-10-21T05:22:27.012284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30455611411,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2320452,"num_read_kb":9167788,"num_write":2316849,"num_write_kb":9267390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18979078144,"data_stored":38582894630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40095,"ondisk_log_size":40095,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47960,"ondisk_log_size":47960,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739382,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11263292,"kb_used_data":9268400,"kb_used_omap":2785,"kb_used_meta":1992094,"kb_avail":83108548,"statfs":{"total":96636764160,"available":85103153152,"internally_reserved":0,"allocated":9490841600,"data_stored":19291726320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2852507,"internal_metadata":2039904613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.89400000000000002,"15min":1.0089999999999999},"min":{"1min":0.26400000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":2.0579999999999998,"5min":4.5659999999999998,"15min":27.184999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85899999999999999,"15min":0.97899999999999998},"min":{"1min":0.254,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.1320000000000001,"5min":5.0620000000000003,"15min":27.251000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sat Oct 21 05:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.90900000000000003,"15min":0.92700000000000005},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.0899999999999999,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.317},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.056,"15min":1.0629999999999999},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.855,"5min":5.7430000000000003,"15min":10.253},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739381,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6055500,"kb_used_data":4552004,"kb_used_omap":1338,"kb_used_meta":1502149,"kb_avail":88316340,"statfs":{"total":96636764160,"available":90435932160,"internally_reserved":0,"allocated":4661252096,"data_stored":9472303556,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1538201579},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,2,11,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.71599999999999997,"15min":0.78100000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":4.5010000000000003,"5min":4.5010000000000003,"15min":6.2140000000000004},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77500000000000002,"15min":0.80400000000000005},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.20899999999999999},"max":{"1min":4.5750000000000002,"5min":4.5750000000000002,"15min":6.0579999999999998},"last":1.347}]},{"osd":2,"last update":"Sat Oct 21 05:23:20 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.77900000000000003,"15min":0.88100000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":4.54,"5min":4.54,"15min":5.0339999999999998},"last":1.272},{"interface":"front","average":{"1min":1.044,"5min":0.81200000000000006,"15min":0.88900000000000001},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":5.1989999999999998,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":1.1559999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6335884,"kb_used_data":4717828,"kb_used_omap":1510,"kb_used_meta":1616537,"kb_avail":88035956,"statfs":{"total":96636764160,"available":90148818944,"internally_reserved":0,"allocated":4831055872,"data_stored":9820453531,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1655334395},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.74199999999999999,"15min":0.81299999999999994},"min":{"1min":0.23300000000000001,"5min":0.189,"15min":0.187},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.76200000000000001,"15min":0.88200000000000001},"min":{"1min":0.253,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":5.8879999999999999,"5min":5.968,"15min":6.4050000000000002},"last":0.86299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:23:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":1.1379999999999999,"15min":0.99099999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.20100000000000001},"max":{"1min":14.053000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.215},{"interface":"front","average":{"1min":1.21,"5min":1.101,"15min":0.96299999999999997},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.831,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.46300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4830052352,"data_stored":9820098560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4659761152,"data_stored":9471623187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9489264640,"data_stored":19291172883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:01.156 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:01.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:24:01.705 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:24:01.728 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:05.447 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:24:05.445+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:24:21.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:22.284 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:24:22.285 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:24:22.305 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2689,"stamp":"2023-10-21T05:24:21.962410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30490674941,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2378225,"num_read_kb":9242058,"num_write":2326417,"num_write_kb":9303443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88243,"ondisk_log_size":88243,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24068476,"kb_used_data":18605220,"kb_used_omap":5634,"kb_used_meta":5457597,"kb_avail":259047044,"statfs":{"total":289910292480,"available":265264173056,"internally_reserved":0,"allocated":19051745280,"data_stored":38653067464,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5769397,"internal_metadata":5588580171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,6,2,11,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17338368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4256,"num_read_kb":16992,"num_write":4230,"num_write_kb":16920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002013"},"pg_stats":[{"pgid":"2.7","version":"150'7787","reported_seq":11385,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.897479+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:24:16.897479+0000","last_peered":"2023-10-21T05:24:16.897479+0000","last_clean":"2023-10-21T05:24:16.897479+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:24:16.897479+0000","last_undegraded":"2023-10-21T05:24:16.897479+0000","last_fullsized":"2023-10-21T05:24:16.897479+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'3984","reported_seq":7827,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:14.742478+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:24:14.742478+0000","last_peered":"2023-10-21T05:24:14.742478+0000","last_clean":"2023-10-21T05:24:14.742478+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:24:14.742478+0000","last_undegraded":"2023-10-21T05:24:14.742478+0000","last_fullsized":"2023-10-21T05:24:14.742478+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'11623","reported_seq":16878,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.897176+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:24:16.897176+0000","last_peered":"2023-10-21T05:24:16.897176+0000","last_clean":"2023-10-21T05:24:16.897176+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:24:16.897176+0000","last_undegraded":"2023-10-21T05:24:16.897176+0000","last_fullsized":"2023-10-21T05:24:16.897176+0000","mapping_epoch":11,"log_start":"57'1600","ondisk_log_start":"57'1600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'7829","reported_seq":12442,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.896839+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:24:16.896839+0000","last_peered":"2023-10-21T05:24:16.896839+0000","last_clean":"2023-10-21T05:24:16.896839+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:24:16.896839+0000","last_undegraded":"2023-10-21T05:24:16.896839+0000","last_fullsized":"2023-10-21T05:24:16.896839+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7829,"log_dups_size":0,"ondisk_log_size":7829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'585400","reported_seq":1157473,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.019729+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:24:17.019729+0000","last_peered":"2023-10-21T05:24:17.019729+0000","last_clean":"2023-10-21T05:24:17.019729+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:24:17.019729+0000","last_undegraded":"2023-10-21T05:24:17.019729+0000","last_fullsized":"2023-10-21T05:24:17.019729+0000","mapping_epoch":14,"log_start":"148'575362","ondisk_log_start":"148'575362","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7540101120,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":576373,"num_read_kb":2277184,"num_write":575245,"num_write_kb":2300980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'15830","reported_seq":22975,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.897426+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:24:16.897426+0000","last_peered":"2023-10-21T05:24:16.897426+0000","last_clean":"2023-10-21T05:24:16.897426+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:24:16.897426+0000","last_undegraded":"2023-10-21T05:24:16.897426+0000","last_fullsized":"2023-10-21T05:24:16.897426+0000","mapping_epoch":11,"log_start":"95'5800","ondisk_log_start":"95'5800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'592356","reported_seq":1172197,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.870500+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:24:17.870500+0000","last_peered":"2023-10-21T05:24:17.870500+0000","last_clean":"2023-10-21T05:24:17.870500+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:24:17.870500+0000","last_undegraded":"2023-10-21T05:24:17.870500+0000","last_fullsized":"2023-10-21T05:24:17.870500+0000","mapping_epoch":14,"log_start":"148'582261","ondisk_log_start":"148'582261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7633907712,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":584179,"num_read_kb":2307544,"num_write":582115,"num_write_kb":2328460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3733,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.905684+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:24:16.905684+0000","last_peered":"2023-10-21T05:24:16.905684+0000","last_clean":"2023-10-21T05:24:16.905684+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:24:16.905684+0000","last_undegraded":"2023-10-21T05:24:16.905684+0000","last_fullsized":"2023-10-21T05:24:16.905684+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'618969","reported_seq":1221422,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:17.013866+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:24:17.013866+0000","last_peered":"2023-10-21T05:24:17.013866+0000","last_clean":"2023-10-21T05:24:17.013866+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:24:17.013866+0000","last_undegraded":"2023-10-21T05:24:17.013866+0000","last_fullsized":"2023-10-21T05:24:17.013866+0000","mapping_epoch":14,"log_start":"148'608954","ondisk_log_start":"148'608954","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7973707776,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":606974,"num_read_kb":2398452,"num_write":608255,"num_write_kb":2433020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7421,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.899063+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:24:16.899063+0000","last_peered":"2023-10-21T05:24:16.899063+0000","last_clean":"2023-10-21T05:24:16.899063+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:24:16.899063+0000","last_undegraded":"2023-10-21T05:24:16.899063+0000","last_fullsized":"2023-10-21T05:24:16.899063+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'566832","reported_seq":1123890,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:17.869040+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:24:17.869040+0000","last_peered":"2023-10-21T05:24:17.869040+0000","last_clean":"2023-10-21T05:24:17.869040+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:24:17.869040+0000","last_undegraded":"2023-10-21T05:24:17.869040+0000","last_fullsized":"2023-10-21T05:24:17.869040+0000","mapping_epoch":14,"log_start":"148'556758","ondisk_log_start":"148'556758","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2843,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 386s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7342362643,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":561240,"num_read_kb":2217808,"num_write":559644,"num_write_kb":2238570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'7746","reported_seq":11276,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:16.897885+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:24:16.897885+0000","last_peered":"2023-10-21T05:24:16.897885+0000","last_clean":"2023-10-21T05:24:16.897885+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:24:16.897885+0000","last_undegraded":"2023-10-21T05:24:16.897885+0000","last_fullsized":"2023-10-21T05:24:16.897885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7746,"log_dups_size":0,"ondisk_log_size":7746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":329,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-21T05:22:54.555252+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:22:54.555252+0000","last_peered":"2023-10-21T05:22:54.555252+0000","last_clean":"2023-10-21T05:22:54.555252+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:22:54.555252+0000","last_undegraded":"2023-10-21T05:22:54.555252+0000","last_fullsized":"2023-10-21T05:22:54.555252+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30490079251,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2328766,"num_read_kb":9200988,"num_write":2325259,"num_write_kb":9301030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19047845888,"data_stored":38651662374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40222,"ondisk_log_size":40222,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":47922,"ondisk_log_size":47922,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739386,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11473000,"kb_used_data":9301792,"kb_used_omap":2785,"kb_used_meta":2168414,"kb_avail":82898840,"statfs":{"total":96636764160,"available":84888412160,"internally_reserved":0,"allocated":9525035008,"data_stored":19325915843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2852507,"internal_metadata":2220456293},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.89400000000000002,"15min":1.0089999999999999},"min":{"1min":0.26400000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":2.0579999999999998,"5min":4.5659999999999998,"15min":27.184999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85899999999999999,"15min":0.97899999999999998},"min":{"1min":0.254,"5min":0.20399999999999999,"15min":0.19900000000000001},"max":{"1min":2.1320000000000001,"5min":5.0620000000000003,"15min":27.251000000000001},"last":1.919}]},{"osd":1,"last update":"Sat Oct 21 05:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.90900000000000003,"15min":0.92700000000000005},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.0899999999999999,"5min":4.7690000000000001,"15min":5.3319999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.056,"15min":1.0629999999999999},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.855,"5min":5.7430000000000003,"15min":10.253},"last":0.45000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739385,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6152268,"kb_used_data":4568452,"kb_used_omap":1338,"kb_used_meta":1582469,"kb_avail":88219572,"statfs":{"total":96636764160,"available":90336841728,"internally_reserved":0,"allocated":4678094848,"data_stored":9489142423,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1620449259},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,9,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.71599999999999997,"15min":0.78100000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":4.5010000000000003,"5min":4.5010000000000003,"15min":6.2140000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77500000000000002,"15min":0.80400000000000005},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.20899999999999999},"max":{"1min":4.5750000000000002,"5min":4.5750000000000002,"15min":6.0579999999999998},"last":1.7749999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:23:20 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.77900000000000003,"15min":0.88100000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":4.54,"5min":4.54,"15min":5.0339999999999998},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.044,"5min":0.81200000000000006,"15min":0.88900000000000001},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":5.1989999999999998,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":1.994}]}]},{"osd":0,"up_from":8,"seq":34359739385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6443208,"kb_used_data":4734976,"kb_used_omap":1510,"kb_used_meta":1706713,"kb_avail":87928632,"statfs":{"total":96636764160,"available":90038919168,"internally_reserved":0,"allocated":4848615424,"data_stored":9838009198,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1747674619},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.74199999999999999,"15min":0.81299999999999994},"min":{"1min":0.23300000000000001,"5min":0.189,"15min":0.187},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.76200000000000001,"15min":0.88200000000000001},"min":{"1min":0.253,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":5.8879999999999999,"5min":5.968,"15min":6.4050000000000002},"last":1.732}]},{"osd":2,"last update":"Sat Oct 21 05:23:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":1.1379999999999999,"15min":0.99099999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.20100000000000001},"max":{"1min":14.053000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.21,"5min":1.101,"15min":0.96299999999999997},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.831,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4847616000,"data_stored":9837662208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4676640768,"data_stored":9488502803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9523589120,"data_stored":19325497363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:22.307 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:22.307 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:23.755 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:24:23.754+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:24:23.756 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:24:23.754+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:24:26.377 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.373+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:24:26.378 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.373+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:24:26.378 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.374+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:24:26.379 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.374+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:24:26.381 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.379+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:24:26.382 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.379+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:24:26.382 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:24:26.383 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:26.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:24:29.997 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:24:29.996+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:24:32.996 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:24:32.997 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:24:32.997 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:24:32.997 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:24:32.998 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:24:32.998 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:24:32.998 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:24:32.999 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:24:32.999 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:24:32.999 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:32.995+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:24:42.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:42.841 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:24:42.841 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:24:42.862 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2699,"stamp":"2023-10-21T05:24:41.965881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30525757181,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2386483,"num_read_kb":9274986,"num_write":2334977,"num_write_kb":9337683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88364,"ondisk_log_size":88364,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23345648,"kb_used_data":18673688,"kb_used_omap":5635,"kb_used_meta":4666300,"kb_avail":259769872,"statfs":{"total":289910292480,"available":266004348928,"internally_reserved":0,"allocated":19121856512,"data_stored":38723178696,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5771077,"internal_metadata":4778291387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,7,4,13,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17502208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4010,"num_read_kb":15976,"num_write":4269,"num_write_kb":17076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002108"},"pg_stats":[{"pgid":"2.7","version":"150'7814","reported_seq":11420,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.377881+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:24:37.377881+0000","last_peered":"2023-10-21T05:24:37.377881+0000","last_clean":"2023-10-21T05:24:37.377881+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:24:37.377881+0000","last_undegraded":"2023-10-21T05:24:37.377881+0000","last_fullsized":"2023-10-21T05:24:37.377881+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7814,"log_dups_size":0,"ondisk_log_size":7814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'3998","reported_seq":7845,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.004084+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:24:37.004084+0000","last_peered":"2023-10-21T05:24:37.004084+0000","last_clean":"2023-10-21T05:24:37.004084+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:24:37.004084+0000","last_undegraded":"2023-10-21T05:24:37.004084+0000","last_fullsized":"2023-10-21T05:24:37.004084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'11664","reported_seq":16931,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.379341+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:24:37.379341+0000","last_peered":"2023-10-21T05:24:37.379341+0000","last_clean":"2023-10-21T05:24:37.379341+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:24:37.379341+0000","last_undegraded":"2023-10-21T05:24:37.379341+0000","last_fullsized":"2023-10-21T05:24:37.379341+0000","mapping_epoch":11,"log_start":"57'1600","ondisk_log_start":"57'1600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'7857","reported_seq":12478,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.377669+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:24:37.377669+0000","last_peered":"2023-10-21T05:24:37.377669+0000","last_clean":"2023-10-21T05:24:37.377669+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:24:37.377669+0000","last_undegraded":"2023-10-21T05:24:37.377669+0000","last_fullsized":"2023-10-21T05:24:37.377669+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'587481","reported_seq":1161541,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.118057+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:24:37.118057+0000","last_peered":"2023-10-21T05:24:37.118057+0000","last_clean":"2023-10-21T05:24:37.118057+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:24:37.118057+0000","last_undegraded":"2023-10-21T05:24:37.118057+0000","last_fullsized":"2023-10-21T05:24:37.118057+0000","mapping_epoch":14,"log_start":"149'577462","ondisk_log_start":"149'577462","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7548637184,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":578361,"num_read_kb":2285108,"num_write":577326,"num_write_kb":2309304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'15883","reported_seq":23040,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:36.268932+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:24:36.268932+0000","last_peered":"2023-10-21T05:24:36.268932+0000","last_clean":"2023-10-21T05:24:36.268932+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:24:36.268932+0000","last_undegraded":"2023-10-21T05:24:36.268932+0000","last_fullsized":"2023-10-21T05:24:36.268932+0000","mapping_epoch":11,"log_start":"95'5800","ondisk_log_start":"95'5800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'594499","reported_seq":1176467,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:38.019494+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:24:38.019494+0000","last_peered":"2023-10-21T05:24:38.019494+0000","last_clean":"2023-10-21T05:24:38.019494+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:24:38.019494+0000","last_undegraded":"2023-10-21T05:24:38.019494+0000","last_fullsized":"2023-10-21T05:24:38.019494+0000","mapping_epoch":14,"log_start":"149'584461","ondisk_log_start":"149'584461","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7642693632,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":586308,"num_read_kb":2316048,"num_write":584258,"num_write_kb":2337032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3739,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:32.264692+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:24:32.264692+0000","last_peered":"2023-10-21T05:24:32.264692+0000","last_clean":"2023-10-21T05:24:32.264692+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:24:32.264692+0000","last_undegraded":"2023-10-21T05:24:32.264692+0000","last_fullsized":"2023-10-21T05:24:32.264692+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'621214","reported_seq":1225785,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.134811+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:24:37.134811+0000","last_peered":"2023-10-21T05:24:37.134811+0000","last_clean":"2023-10-21T05:24:37.134811+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:24:37.134811+0000","last_undegraded":"2023-10-21T05:24:37.134811+0000","last_fullsized":"2023-10-21T05:24:37.134811+0000","mapping_epoch":14,"log_start":"149'611154","ondisk_log_start":"149'611154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7982907392,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":609094,"num_read_kb":2406884,"num_write":610500,"num_write_kb":2442000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7433,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:32.261375+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:24:32.261375+0000","last_peered":"2023-10-21T05:24:32.261375+0000","last_clean":"2023-10-21T05:24:32.261375+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:24:32.261375+0000","last_undegraded":"2023-10-21T05:24:32.261375+0000","last_fullsized":"2023-10-21T05:24:32.261375+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'568993","reported_seq":1128085,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:38.017251+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:24:38.017251+0000","last_peered":"2023-10-21T05:24:38.017251+0000","last_clean":"2023-10-21T05:24:38.017251+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:24:38.017251+0000","last_undegraded":"2023-10-21T05:24:38.017251+0000","last_fullsized":"2023-10-21T05:24:38.017251+0000","mapping_epoch":14,"log_start":"149'558958","ondisk_log_start":"149'558958","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":2931,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 406s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7350923283,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":563261,"num_read_kb":2225876,"num_write":561735,"num_write_kb":2246934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'7774","reported_seq":11312,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:37.376856+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:24:37.376856+0000","last_peered":"2023-10-21T05:24:37.376856+0000","last_clean":"2023-10-21T05:24:37.376856+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:24:37.376856+0000","last_undegraded":"2023-10-21T05:24:37.376856+0000","last_fullsized":"2023-10-21T05:24:37.376856+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":330,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.568477+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:24:04.568477+0000","last_peered":"2023-10-21T05:24:04.568477+0000","last_clean":"2023-10-21T05:24:04.568477+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:24:04.568477+0000","last_undegraded":"2023-10-21T05:24:04.568477+0000","last_fullsized":"2023-10-21T05:24:04.568477+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30525161491,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2337024,"num_read_kb":9233916,"num_write":2333819,"num_write_kb":9335270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19117916160,"data_stored":38721732646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40152,"ondisk_log_size":40152,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":48113,"ondisk_log_size":48113,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739390,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10527624,"kb_used_data":9336128,"kb_used_omap":2787,"kb_used_meta":1188700,"kb_avail":83844216,"statfs":{"total":96636764160,"available":85856477184,"internally_reserved":0,"allocated":9560195072,"data_stored":19361075907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2854187,"internal_metadata":1217229525},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.98399999999999999},"min":{"1min":0.27600000000000002,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":5.5609999999999999,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.113,"5min":0.86099999999999999,"15min":0.99099999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":5.5229999999999997,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:24:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.89800000000000002,"15min":0.91000000000000003},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.026,"5min":0.99399999999999999,"15min":1.0029999999999999},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":0.38100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739389,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6267220,"kb_used_data":4585228,"kb_used_omap":1338,"kb_used_meta":1680645,"kb_avail":88104620,"statfs":{"total":96636764160,"available":90219130880,"internally_reserved":0,"allocated":4695273472,"data_stored":9506321047,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1720981483},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4,12,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.67400000000000004,"15min":0.75800000000000001},"min":{"1min":0.20499999999999999,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":1.8400000000000001,"5min":4.5010000000000003,"15min":6.0899999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.81499999999999995,"15min":0.81999999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20899999999999999},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.43099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78100000000000003,"15min":0.88800000000000001},"min":{"1min":0.26400000000000001,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":2.5209999999999999,"5min":4.54,"15min":5.0339999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.86499999999999999,"15min":0.91000000000000003},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":3.5990000000000002,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6550804,"kb_used_data":4752332,"kb_used_omap":1510,"kb_used_meta":1796953,"kb_avail":87821036,"statfs":{"total":96636764160,"available":89928740864,"internally_reserved":0,"allocated":4866387968,"data_stored":9855781742,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1840080379},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":2.2250000000000001,"5min":5.923,"15min":5.923},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.82799999999999996,"15min":0.85899999999999999},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":2.5590000000000002,"5min":5.968,"15min":6.4050000000000002},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.1020000000000001,"15min":0.98899999999999999},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.20100000000000001},"max":{"1min":3.7999999999999998,"5min":14.053000000000001,"15min":14.053000000000001},"last":4.867},{"interface":"front","average":{"1min":1.02,"5min":1.1299999999999999,"15min":0.98299999999999998},"min":{"1min":0.23999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.21,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4865343488,"data_stored":9855389696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4693983232,"data_stored":9505845267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9558589440,"data_stored":19360497683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:42.864 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:42.865 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:42.965 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.963+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:24:42.965 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.963+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:24:42.966 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.963+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:24:42.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.965+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:24:42.967 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.965+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:24:42.972 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:24:42.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.970+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:24:42.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:24:42.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.970+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:24:42.973 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:42.970+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:24:50.678 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.667+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:24:50.679 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.667+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:24:50.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.678+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:24:50.681 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.678+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:24:50.682 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.678+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:24:50.689 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.687+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:24:50.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.687+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:24:50.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.687+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:24:50.690 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.687+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:24:50.691 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:50.687+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:24:55.739 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:24:55.735+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:24:55.739 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:24:55.735+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:24:58.100 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.092+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:24:58.100 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.092+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:24:58.100 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.095+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:24:58.114 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.108+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:24:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.108+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:24:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.111+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:24:58.115 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.111+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:24:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.111+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:24:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.111+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:24:58.116 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:24:58.111+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:25:02.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:03.383 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:25:03.384 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:25:03.403 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2709,"stamp":"2023-10-21T05:25:01.969336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30560798461,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2394793,"num_read_kb":9308146,"num_write":2343523,"num_write_kb":9371867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88428,"ondisk_log_size":88428,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23765708,"kb_used_data":18743024,"kb_used_omap":5635,"kb_used_meta":5017020,"kb_avail":259349812,"statfs":{"total":289910292480,"available":265574207488,"internally_reserved":0,"allocated":19192856576,"data_stored":38794178760,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5771077,"internal_metadata":5137428667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,10,9,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17428480,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4168,"num_read_kb":16636,"num_write":4248,"num_write_kb":16992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002068"},"pg_stats":[{"pgid":"2.7","version":"150'7836","reported_seq":11451,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.117494+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:24:58.117494+0000","last_peered":"2023-10-21T05:24:58.117494+0000","last_clean":"2023-10-21T05:24:58.117494+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:24:58.117494+0000","last_undegraded":"2023-10-21T05:24:58.117494+0000","last_fullsized":"2023-10-21T05:24:58.117494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7836,"log_dups_size":0,"ondisk_log_size":7836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'4009","reported_seq":7860,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:57.856727+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:24:57.856727+0000","last_peered":"2023-10-21T05:24:57.856727+0000","last_clean":"2023-10-21T05:24:57.856727+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:24:57.856727+0000","last_undegraded":"2023-10-21T05:24:57.856727+0000","last_fullsized":"2023-10-21T05:24:57.856727+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'11697","reported_seq":16976,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.122932+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:24:58.122932+0000","last_peered":"2023-10-21T05:24:58.122932+0000","last_clean":"2023-10-21T05:24:58.122932+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:24:58.122932+0000","last_undegraded":"2023-10-21T05:24:58.122932+0000","last_fullsized":"2023-10-21T05:24:58.122932+0000","mapping_epoch":11,"log_start":"57'1600","ondisk_log_start":"57'1600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'7879","reported_seq":12508,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.110218+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:24:58.110218+0000","last_peered":"2023-10-21T05:24:58.110218+0000","last_clean":"2023-10-21T05:24:58.110218+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:24:58.110218+0000","last_undegraded":"2023-10-21T05:24:58.110218+0000","last_fullsized":"2023-10-21T05:24:58.110218+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7879,"log_dups_size":0,"ondisk_log_size":7879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'589672","reported_seq":1165794,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:57.217639+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:24:57.217639+0000","last_peered":"2023-10-21T05:24:57.217639+0000","last_clean":"2023-10-21T05:24:57.217639+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:24:57.217639+0000","last_undegraded":"2023-10-21T05:24:57.217639+0000","last_fullsized":"2023-10-21T05:24:57.217639+0000","mapping_epoch":14,"log_start":"149'579662","ondisk_log_start":"149'579662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7557611520,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":580424,"num_read_kb":2293336,"num_write":579517,"num_write_kb":2318068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'15907","reported_seq":23080,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:56.954671+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:24:56.954671+0000","last_peered":"2023-10-21T05:24:56.954671+0000","last_clean":"2023-10-21T05:24:56.954671+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:24:56.954671+0000","last_undegraded":"2023-10-21T05:24:56.954671+0000","last_fullsized":"2023-10-21T05:24:56.954671+0000","mapping_epoch":11,"log_start":"97'5900","ondisk_log_start":"97'5900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'596638","reported_seq":1180674,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.197252+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:24:58.197252+0000","last_peered":"2023-10-21T05:24:58.197252+0000","last_clean":"2023-10-21T05:24:58.197252+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:24:58.197252+0000","last_undegraded":"2023-10-21T05:24:58.197252+0000","last_fullsized":"2023-10-21T05:24:58.197252+0000","mapping_epoch":14,"log_start":"149'586561","ondisk_log_start":"149'586561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7651459072,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":588375,"num_read_kb":2324300,"num_write":586397,"num_write_kb":2345588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3747,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:52.740892+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:24:52.740892+0000","last_peered":"2023-10-21T05:24:52.740892+0000","last_clean":"2023-10-21T05:24:52.740892+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:24:52.740892+0000","last_undegraded":"2023-10-21T05:24:52.740892+0000","last_fullsized":"2023-10-21T05:24:52.740892+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'623378","reported_seq":1230109,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:57.206973+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:24:57.206973+0000","last_peered":"2023-10-21T05:24:57.206973+0000","last_clean":"2023-10-21T05:24:57.206973+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:24:57.206973+0000","last_undegraded":"2023-10-21T05:24:57.206973+0000","last_fullsized":"2023-10-21T05:24:57.206973+0000","mapping_epoch":14,"log_start":"149'613354","ondisk_log_start":"149'613354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":7991775232,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":611251,"num_read_kb":2415476,"num_write":612664,"num_write_kb":2450656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7449,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:52.739558+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:24:52.739558+0000","last_peered":"2023-10-21T05:24:52.739558+0000","last_clean":"2023-10-21T05:24:52.739558+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:24:52.739558+0000","last_undegraded":"2023-10-21T05:24:52.739558+0000","last_fullsized":"2023-10-21T05:24:52.739558+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'571129","reported_seq":1132252,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:58.211211+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:24:58.211211+0000","last_peered":"2023-10-21T05:24:58.211211+0000","last_clean":"2023-10-21T05:24:58.211211+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:24:58.211211+0000","last_undegraded":"2023-10-21T05:24:58.211211+0000","last_fullsized":"2023-10-21T05:24:58.211211+0000","mapping_epoch":14,"log_start":"149'561058","ondisk_log_start":"149'561058","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3031,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 426s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7359356947,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":565284,"num_read_kb":2233964,"num_write":563787,"num_write_kb":2255142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'7796","reported_seq":11342,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:58.129690+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:24:58.129690+0000","last_peered":"2023-10-21T05:24:58.129690+0000","last_clean":"2023-10-21T05:24:58.129690+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:24:58.129690+0000","last_undegraded":"2023-10-21T05:24:58.129690+0000","last_fullsized":"2023-10-21T05:24:58.129690+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":330,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.568477+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:24:04.568477+0000","last_peered":"2023-10-21T05:24:04.568477+0000","last_clean":"2023-10-21T05:24:04.568477+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:24:04.568477+0000","last_undegraded":"2023-10-21T05:24:04.568477+0000","last_fullsized":"2023-10-21T05:24:04.568477+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30560202771,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2345334,"num_read_kb":9267076,"num_write":2342365,"num_write_kb":9369454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19187875840,"data_stored":38791692326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40182,"ondisk_log_size":40182,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":48147,"ondisk_log_size":48147,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739394,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10727652,"kb_used_data":9370012,"kb_used_omap":2787,"kb_used_meta":1354844,"kb_avail":83644188,"statfs":{"total":96636764160,"available":85651648512,"internally_reserved":0,"allocated":9594892288,"data_stored":19395773123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2854187,"internal_metadata":1387360981},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.98399999999999999},"min":{"1min":0.27600000000000002,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":5.5609999999999999,"5min":5.5609999999999999,"15min":27.184999999999999},"last":3.1070000000000002},{"interface":"front","average":{"1min":1.113,"5min":0.86099999999999999,"15min":0.99099999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":5.5229999999999997,"5min":5.5229999999999997,"15min":27.251000000000001},"last":3.1280000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:24:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.89800000000000002,"15min":0.91000000000000003},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.026,"5min":0.99399999999999999,"15min":1.0029999999999999},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739393,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6379092,"kb_used_data":4602696,"kb_used_omap":1338,"kb_used_meta":1775045,"kb_avail":87992748,"statfs":{"total":96636764160,"available":90104573952,"internally_reserved":0,"allocated":4713160704,"data_stored":9524208279,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1817647083},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,4,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.67400000000000004,"15min":0.75800000000000001},"min":{"1min":0.20499999999999999,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":1.8400000000000001,"5min":4.5010000000000003,"15min":6.0899999999999999},"last":2.1709999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.81499999999999995,"15min":0.81999999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20899999999999999},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":2.198}]},{"osd":2,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78100000000000003,"15min":0.88800000000000001},"min":{"1min":0.26400000000000001,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":2.5209999999999999,"5min":4.54,"15min":5.0339999999999998},"last":2.1110000000000002},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.86499999999999999,"15min":0.91000000000000003},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":3.5990000000000002,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":1.8560000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6658964,"kb_used_data":4770316,"kb_used_omap":1510,"kb_used_meta":1887129,"kb_avail":87712876,"statfs":{"total":96636764160,"available":89817985024,"internally_reserved":0,"allocated":4884803584,"data_stored":9874197358,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":1932420603},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":2.2250000000000001,"5min":5.923,"15min":5.923},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.82799999999999996,"15min":0.85899999999999999},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":2.5590000000000002,"5min":5.968,"15min":6.4050000000000002},"last":0.83499999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.1020000000000001,"15min":0.98899999999999999},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.20100000000000001},"max":{"1min":3.7999999999999998,"5min":14.053000000000001,"15min":14.053000000000001},"last":1.165},{"interface":"front","average":{"1min":1.02,"5min":1.1299999999999999,"15min":0.98299999999999998},"min":{"1min":0.23999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.21,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4883173376,"data_stored":9873219584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4711149568,"data_stored":9523011603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9593552896,"data_stored":19395461139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:03.406 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:03.406 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:23.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:23.944 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:25:23.945 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:25:23.966 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2720,"stamp":"2023-10-21T05:25:21.973810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30595598077,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2403370,"num_read_kb":9342398,"num_write":2352008,"num_write_kb":9405807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88526,"ondisk_log_size":88526,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24159288,"kb_used_data":18809884,"kb_used_omap":5635,"kb_used_meta":5343740,"kb_avail":258956232,"statfs":{"total":289910292480,"available":265171181568,"internally_reserved":0,"allocated":19261321216,"data_stored":38862631616,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5771077,"internal_metadata":5471989947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,7,7,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17858560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4365,"num_read_kb":17416,"num_write":4350,"num_write_kb":17400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001709"},"pg_stats":[{"pgid":"2.7","version":"150'7866","reported_seq":11488,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.497192+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:25:13.497192+0000","last_peered":"2023-10-21T05:25:13.497192+0000","last_clean":"2023-10-21T05:25:13.497192+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:25:13.497192+0000","last_undegraded":"2023-10-21T05:25:13.497192+0000","last_fullsized":"2023-10-21T05:25:13.497192+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'4024","reported_seq":7879,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.494563+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:25:13.494563+0000","last_peered":"2023-10-21T05:25:13.494563+0000","last_clean":"2023-10-21T05:25:13.494563+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:25:13.494563+0000","last_undegraded":"2023-10-21T05:25:13.494563+0000","last_fullsized":"2023-10-21T05:25:13.494563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'11742","reported_seq":17031,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.496566+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:25:13.496566+0000","last_peered":"2023-10-21T05:25:13.496566+0000","last_clean":"2023-10-21T05:25:13.496566+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:25:13.496566+0000","last_undegraded":"2023-10-21T05:25:13.496566+0000","last_fullsized":"2023-10-21T05:25:13.496566+0000","mapping_epoch":11,"log_start":"62'1700","ondisk_log_start":"62'1700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'7909","reported_seq":12545,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.496284+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:25:13.496284+0000","last_peered":"2023-10-21T05:25:13.496284+0000","last_clean":"2023-10-21T05:25:13.496284+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:25:13.496284+0000","last_undegraded":"2023-10-21T05:25:13.496284+0000","last_fullsized":"2023-10-21T05:25:13.496284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7909,"log_dups_size":0,"ondisk_log_size":7909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'591743","reported_seq":1169985,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:17.356516+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:25:17.356516+0000","last_peered":"2023-10-21T05:25:17.356516+0000","last_clean":"2023-10-21T05:25:17.356516+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:25:17.356516+0000","last_undegraded":"2023-10-21T05:25:17.356516+0000","last_fullsized":"2023-10-21T05:25:17.356516+0000","mapping_epoch":14,"log_start":"149'581662","ondisk_log_start":"149'581662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7566077952,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":582541,"num_read_kb":2301788,"num_write":581588,"num_write_kb":2326352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'15943","reported_seq":23133,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:16.806323+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:25:16.806323+0000","last_peered":"2023-10-21T05:25:16.806323+0000","last_clean":"2023-10-21T05:25:16.806323+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:25:16.806323+0000","last_undegraded":"2023-10-21T05:25:16.806323+0000","last_fullsized":"2023-10-21T05:25:16.806323+0000","mapping_epoch":11,"log_start":"97'5900","ondisk_log_start":"97'5900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'598795","reported_seq":1184996,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:18.284773+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:25:18.284773+0000","last_peered":"2023-10-21T05:25:18.284773+0000","last_clean":"2023-10-21T05:25:18.284773+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:25:18.284773+0000","last_undegraded":"2023-10-21T05:25:18.284773+0000","last_fullsized":"2023-10-21T05:25:18.284773+0000","mapping_epoch":14,"log_start":"149'588761","ondisk_log_start":"149'588761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7660331008,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":590536,"num_read_kb":2332924,"num_write":588554,"num_write_kb":2354216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3756,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.217999+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:25:13.217999+0000","last_peered":"2023-10-21T05:25:13.217999+0000","last_clean":"2023-10-21T05:25:13.217999+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:25:13.217999+0000","last_undegraded":"2023-10-21T05:25:13.217999+0000","last_fullsized":"2023-10-21T05:25:13.217999+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'625617","reported_seq":1234598,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:17.339415+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:25:17.339415+0000","last_peered":"2023-10-21T05:25:17.339415+0000","last_clean":"2023-10-21T05:25:17.339415+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:25:17.339415+0000","last_undegraded":"2023-10-21T05:25:17.339415+0000","last_fullsized":"2023-10-21T05:25:17.339415+0000","mapping_epoch":14,"log_start":"149'615554","ondisk_log_start":"149'615554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8000966656,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":613502,"num_read_kb":2424464,"num_write":614903,"num_write_kb":2459612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7466,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.218253+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:25:13.218253+0000","last_peered":"2023-10-21T05:25:13.218253+0000","last_clean":"2023-10-21T05:25:13.218253+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:25:13.218253+0000","last_undegraded":"2023-10-21T05:25:13.218253+0000","last_fullsized":"2023-10-21T05:25:13.218253+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'573274","reported_seq":1136457,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:18.285649+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:25:18.285649+0000","last_peered":"2023-10-21T05:25:18.285649+0000","last_clean":"2023-10-21T05:25:18.285649+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:25:18.285649+0000","last_undegraded":"2023-10-21T05:25:18.285649+0000","last_fullsized":"2023-10-21T05:25:18.285649+0000","mapping_epoch":14,"log_start":"149'563258","ondisk_log_start":"149'563258","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3183,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 446s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7367626771,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":567332,"num_read_kb":2242152,"num_write":565805,"num_write_kb":2263214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'7826","reported_seq":11379,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:25:13.496011+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:25:13.496011+0000","last_peered":"2023-10-21T05:25:13.496011+0000","last_clean":"2023-10-21T05:25:13.496011+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:25:13.496011+0000","last_undegraded":"2023-10-21T05:25:13.496011+0000","last_fullsized":"2023-10-21T05:25:13.496011+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":331,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:27.922031+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:24:27.922031+0000","last_peered":"2023-10-21T05:24:27.922031+0000","last_clean":"2023-10-21T05:24:27.922031+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:24:27.922031+0000","last_undegraded":"2023-10-21T05:24:27.922031+0000","last_fullsized":"2023-10-21T05:24:27.922031+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30595002387,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2353911,"num_read_kb":9301328,"num_write":2350850,"num_write_kb":9403394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19257196544,"data_stored":38861013030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40194,"ondisk_log_size":40194,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68134,"internal_metadata":0},"log_size":48233,"ondisk_log_size":48233,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739398,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10905940,"kb_used_data":9403916,"kb_used_omap":2787,"kb_used_meta":1499228,"kb_avail":83465900,"statfs":{"total":96636764160,"available":85469081600,"internally_reserved":0,"allocated":9629609984,"data_stored":19430486891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2854187,"internal_metadata":1535210197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.98399999999999999},"min":{"1min":0.27600000000000002,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":5.5609999999999999,"5min":5.5609999999999999,"15min":27.184999999999999},"last":1.038},{"interface":"front","average":{"1min":1.113,"5min":0.86099999999999999,"15min":0.99099999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":5.5229999999999997,"5min":5.5229999999999997,"15min":27.251000000000001},"last":1.0580000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:24:26 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.89800000000000002,"15min":0.91000000000000003},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.995},{"interface":"front","average":{"1min":1.026,"5min":0.99399999999999999,"15min":1.0029999999999999},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739397,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6474948,"kb_used_data":4618620,"kb_used_omap":1338,"kb_used_meta":1854981,"kb_avail":87896892,"statfs":{"total":96636764160,"available":90006417408,"internally_reserved":0,"allocated":4729466880,"data_stored":9540510527,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1899501547},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.67400000000000004,"15min":0.75800000000000001},"min":{"1min":0.20499999999999999,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":1.8400000000000001,"5min":4.5010000000000003,"15min":6.0899999999999999},"last":0.313},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.81499999999999995,"15min":0.81999999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20899999999999999},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":1.9079999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.78100000000000003,"15min":0.88800000000000001},"min":{"1min":0.26400000000000001,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":2.5209999999999999,"5min":4.54,"15min":5.0339999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.86499999999999999,"15min":0.91000000000000003},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":3.5990000000000002,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739397,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6778400,"kb_used_data":4787348,"kb_used_omap":1510,"kb_used_meta":1989529,"kb_avail":87593440,"statfs":{"total":96636764160,"available":89695682560,"internally_reserved":0,"allocated":4902244352,"data_stored":9891634198,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1546757,"internal_metadata":2037278203},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.79500000000000004,"15min":0.80600000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":2.2250000000000001,"5min":5.923,"15min":5.923},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.82799999999999996,"15min":0.85899999999999999},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":2.5590000000000002,"5min":5.968,"15min":6.4050000000000002},"last":0.28599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.1020000000000001,"15min":0.98899999999999999},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.20100000000000001},"max":{"1min":3.7999999999999998,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.02,"5min":1.1299999999999999,"15min":0.98299999999999998},"min":{"1min":0.23999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.21,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.99199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4900847616,"data_stored":9890893824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4728250368,"data_stored":9540112403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9628098560,"data_stored":19430006803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:23.968 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:23.968 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:34.978 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:34.975+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:25:34.979 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:34.975+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:25:34.983 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:34.981+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:25:34.984 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:34.981+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:25:41.382 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:41.379+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:25:41.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:41.390+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:25:42.563 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:42.557+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:25:42.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:42.557+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:25:42.565 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:42.562+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:25:42.567 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:42.564+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:25:43.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:44.392 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.390+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:25:44.393 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.390+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:25:44.408 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.402+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:25:44.408 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.402+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:25:44.408 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.404+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:25:44.409 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.404+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:25:44.409 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.404+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:25:44.409 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.404+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:25:44.410 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:25:44.404+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:25:44.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:25:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:25:44.518 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2731,"stamp":"2023-10-21T05:25:43.977599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30639662845,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2414012,"num_read_kb":9384854,"num_write":2362735,"num_write_kb":9448715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88590,"ondisk_log_size":88590,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23042644,"kb_used_data":18896248,"kb_used_omap":5625,"kb_used_meta":4140742,"kb_avail":260072876,"statfs":{"total":289910292480,"available":266314625024,"internally_reserved":0,"allocated":19349757952,"data_stored":38951068352,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5760129,"internal_metadata":4240120703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,26,9,44,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26243072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6300,"num_read_kb":25132,"num_write":6394,"num_write_kb":25576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002034"},"pg_stats":[{"pgid":"2.7","version":"151'7896","reported_seq":11528,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.922389+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:25:42.922389+0000","last_peered":"2023-10-21T05:25:42.922389+0000","last_clean":"2023-10-21T05:25:42.922389+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:25:42.922389+0000","last_undegraded":"2023-10-21T05:25:42.922389+0000","last_fullsized":"2023-10-21T05:25:42.922389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'4040","reported_seq":7901,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.563475+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:25:42.563475+0000","last_peered":"2023-10-21T05:25:42.563475+0000","last_clean":"2023-10-21T05:25:42.563475+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:25:42.563475+0000","last_undegraded":"2023-10-21T05:25:42.563475+0000","last_fullsized":"2023-10-21T05:25:42.563475+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'11786","reported_seq":17090,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.933623+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:25:42.933623+0000","last_peered":"2023-10-21T05:25:42.933623+0000","last_clean":"2023-10-21T05:25:42.933623+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:25:42.933623+0000","last_undegraded":"2023-10-21T05:25:42.933623+0000","last_fullsized":"2023-10-21T05:25:42.933623+0000","mapping_epoch":11,"log_start":"62'1700","ondisk_log_start":"62'1700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'7939","reported_seq":12585,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.923385+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:25:42.923385+0000","last_peered":"2023-10-21T05:25:42.923385+0000","last_clean":"2023-10-21T05:25:42.923385+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:25:42.923385+0000","last_undegraded":"2023-10-21T05:25:42.923385+0000","last_fullsized":"2023-10-21T05:25:42.923385+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'594470","reported_seq":1175431,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.464037+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:25:42.464037+0000","last_peered":"2023-10-21T05:25:42.464037+0000","last_clean":"2023-10-21T05:25:42.464037+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:25:42.464037+0000","last_undegraded":"2023-10-21T05:25:42.464037+0000","last_fullsized":"2023-10-21T05:25:42.464037+0000","mapping_epoch":14,"log_start":"150'584462","ondisk_log_start":"150'584462","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7577255936,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":585261,"num_read_kb":2312624,"num_write":584315,"num_write_kb":2337260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'16015","reported_seq":23225,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.061121+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:25:42.061121+0000","last_peered":"2023-10-21T05:25:42.061121+0000","last_clean":"2023-10-21T05:25:42.061121+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:25:42.061121+0000","last_undegraded":"2023-10-21T05:25:42.061121+0000","last_fullsized":"2023-10-21T05:25:42.061121+0000","mapping_epoch":11,"log_start":"97'6000","ondisk_log_start":"97'6000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'601495","reported_seq":1190376,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:43.470734+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:25:43.470734+0000","last_peered":"2023-10-21T05:25:43.470734+0000","last_clean":"2023-10-21T05:25:43.470734+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:25:43.470734+0000","last_undegraded":"2023-10-21T05:25:43.470734+0000","last_fullsized":"2023-10-21T05:25:43.470734+0000","mapping_epoch":14,"log_start":"150'591461","ondisk_log_start":"150'591461","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7671410688,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":593218,"num_read_kb":2343608,"num_write":591254,"num_write_kb":2365016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3767,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:38.819369+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:25:38.819369+0000","last_peered":"2023-10-21T05:25:38.819369+0000","last_clean":"2023-10-21T05:25:38.819369+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:25:38.819369+0000","last_undegraded":"2023-10-21T05:25:38.819369+0000","last_fullsized":"2023-10-21T05:25:38.819369+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'628394","reported_seq":1240116,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.465389+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:25:42.465389+0000","last_peered":"2023-10-21T05:25:42.465389+0000","last_clean":"2023-10-21T05:25:42.465389+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:25:42.465389+0000","last_undegraded":"2023-10-21T05:25:42.465389+0000","last_fullsized":"2023-10-21T05:25:42.465389+0000","mapping_epoch":14,"log_start":"150'618354","ondisk_log_start":"150'618354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8012406784,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":616240,"num_read_kb":2435404,"num_write":617680,"num_write_kb":2470720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7486,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:38.822359+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:25:38.822359+0000","last_peered":"2023-10-21T05:25:38.822359+0000","last_clean":"2023-10-21T05:25:38.822359+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:25:38.822359+0000","last_undegraded":"2023-10-21T05:25:38.822359+0000","last_fullsized":"2023-10-21T05:25:38.822359+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'575915","reported_seq":1141614,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:43.471161+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:25:43.471161+0000","last_peered":"2023-10-21T05:25:43.471161+0000","last_clean":"2023-10-21T05:25:43.471161+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:25:43.471161+0000","last_undegraded":"2023-10-21T05:25:43.471161+0000","last_fullsized":"2023-10-21T05:25:43.471161+0000","mapping_epoch":14,"log_start":"150'565859","ondisk_log_start":"150'565859","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3329,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 472s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7377993747,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":569834,"num_read_kb":2252148,"num_write":568328,"num_write_kb":2273306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'7854","reported_seq":11417,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:42.931717+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:25:42.931717+0000","last_peered":"2023-10-21T05:25:42.931717+0000","last_clean":"2023-10-21T05:25:42.931717+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:25:42.931717+0000","last_undegraded":"2023-10-21T05:25:42.931717+0000","last_fullsized":"2023-10-21T05:25:42.931717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7854,"log_dups_size":0,"ondisk_log_size":7854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":331,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:27.922031+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:24:27.922031+0000","last_peered":"2023-10-21T05:24:27.922031+0000","last_clean":"2023-10-21T05:24:27.922031+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:24:27.922031+0000","last_undegraded":"2023-10-21T05:24:27.922031+0000","last_fullsized":"2023-10-21T05:24:27.922031+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30639067155,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2364553,"num_read_kb":9343784,"num_write":2361577,"num_write_kb":9446302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19345240064,"data_stored":38949056550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40138,"ondisk_log_size":40138,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63820,"internal_metadata":0},"log_size":48353,"ondisk_log_size":48353,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739403,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11145700,"kb_used_data":9446616,"kb_used_omap":2787,"kb_used_meta":1696284,"kb_avail":83226140,"statfs":{"total":96636764160,"available":85223567360,"internally_reserved":0,"allocated":9673334784,"data_stored":19474211691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2854187,"internal_metadata":1736995541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:27 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.88400000000000001,"15min":1.01},"min":{"1min":0.24099999999999999,"5min":0.215,"15min":0.186},"max":{"1min":3.1070000000000002,"5min":5.5609999999999999,"15min":27.184999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.88100000000000001,"15min":0.98799999999999999},"min":{"1min":0.26800000000000002,"5min":0.191,"15min":0.191},"max":{"1min":3.1280000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.89800000000000002,"15min":0.90900000000000003},"min":{"1min":0.26700000000000002,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.7330000000000001,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":1.216},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":0.99199999999999999},"min":{"1min":0.29599999999999999,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":2.6989999999999998,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":1.054}]}]},{"osd":1,"up_from":8,"seq":34359739402,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6592376,"kb_used_data":4640240,"kb_used_omap":1338,"kb_used_meta":1950789,"kb_avail":87779464,"statfs":{"total":96636764160,"available":89886171136,"internally_reserved":0,"allocated":4751605760,"data_stored":9562649407,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1997608939},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,41,11,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.74299999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":2.6030000000000002,"5min":4.5010000000000003,"15min":6.0899999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.524,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.47899999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.80800000000000005,"15min":0.88600000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":4.6289999999999996,"5min":4.6289999999999996,"15min":5.0339999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.90400000000000003,"15min":0.89300000000000002},"min":{"1min":0.222,"5min":0.22,"15min":0.22},"max":{"1min":3.621,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":0.51600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5304568,"kb_used_data":4809392,"kb_used_omap":1499,"kb_used_meta":493668,"kb_avail":89067272,"statfs":{"total":96636764160,"available":91204886528,"internally_reserved":0,"allocated":4924817408,"data_stored":9914207254,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":505516223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,25,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.78000000000000003},"min":{"1min":0.215,"5min":0.187,"15min":0.187},"max":{"1min":1.6060000000000001,"5min":5.923,"15min":5.923},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.71399999999999997,"15min":0.82499999999999996},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":1.1699999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.1339999999999999,"15min":0.98199999999999998},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":4.867,"5min":14.053000000000001,"15min":14.053000000000001},"last":2.6059999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.0529999999999999,"15min":0.96899999999999997},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":3.173,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4923371520,"data_stored":9913417728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4749647872,"data_stored":9561509907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9672220672,"data_stored":19474128915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:44.520 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:44.521 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:47.080 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:25:47.077+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:25:47.081 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:25:47.077+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:25:47.081 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:25:47.077+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:25:47.081 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:25:47.077+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:25:54.956 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:25:54.954+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:26:04.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:26:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:26:05.055 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:26:05.077 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2741,"stamp":"2023-10-21T05:26:03.983223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30674499325,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2422386,"num_read_kb":9418318,"num_write":2371235,"num_write_kb":9482715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88607,"ondisk_log_size":88607,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23432900,"kb_used_data":18963880,"kb_used_omap":5625,"kb_used_meta":4463366,"kb_avail":259682620,"statfs":{"total":289910292480,"available":265915002880,"internally_reserved":0,"allocated":19419013120,"data_stored":39020323520,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5760129,"internal_metadata":4570487679},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26337280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6304,"num_read_kb":25196,"num_write":6425,"num_write_kb":25700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004230"},"pg_stats":[{"pgid":"2.7","version":"151'7916","reported_seq":11556,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.707212+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:25:59.707212+0000","last_peered":"2023-10-21T05:25:59.707212+0000","last_clean":"2023-10-21T05:25:59.707212+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:25:59.707212+0000","last_undegraded":"2023-10-21T05:25:59.707212+0000","last_fullsized":"2023-10-21T05:25:59.707212+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7916,"log_dups_size":0,"ondisk_log_size":7916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'4050","reported_seq":7915,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.298236+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:25:59.298236+0000","last_peered":"2023-10-21T05:25:59.298236+0000","last_clean":"2023-10-21T05:25:59.298236+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:25:59.298236+0000","last_undegraded":"2023-10-21T05:25:59.298236+0000","last_fullsized":"2023-10-21T05:25:59.298236+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'11816","reported_seq":17133,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.726212+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:25:59.726212+0000","last_peered":"2023-10-21T05:25:59.726212+0000","last_clean":"2023-10-21T05:25:59.726212+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:25:59.726212+0000","last_undegraded":"2023-10-21T05:25:59.726212+0000","last_fullsized":"2023-10-21T05:25:59.726212+0000","mapping_epoch":11,"log_start":"62'1800","ondisk_log_start":"62'1800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'7959","reported_seq":12614,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.707240+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:25:59.707240+0000","last_peered":"2023-10-21T05:25:59.707240+0000","last_clean":"2023-10-21T05:25:59.707240+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:25:59.707240+0000","last_undegraded":"2023-10-21T05:25:59.707240+0000","last_fullsized":"2023-10-21T05:25:59.707240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'596565","reported_seq":1179657,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.579672+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:26:02.579672+0000","last_peered":"2023-10-21T05:26:02.579672+0000","last_clean":"2023-10-21T05:26:02.579672+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:26:02.579672+0000","last_undegraded":"2023-10-21T05:26:02.579672+0000","last_fullsized":"2023-10-21T05:26:02.579672+0000","mapping_epoch":14,"log_start":"150'586562","ondisk_log_start":"150'586562","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7585845248,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":587390,"num_read_kb":2321128,"num_write":586410,"num_write_kb":2345640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'16059","reported_seq":23286,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.297103+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:25:59.297103+0000","last_peered":"2023-10-21T05:25:59.297103+0000","last_clean":"2023-10-21T05:25:59.297103+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:25:59.297103+0000","last_undegraded":"2023-10-21T05:25:59.297103+0000","last_fullsized":"2023-10-21T05:25:59.297103+0000","mapping_epoch":11,"log_start":"97'6000","ondisk_log_start":"97'6000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'603623","reported_seq":1194616,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:03.551866+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:26:03.551866+0000","last_peered":"2023-10-21T05:26:03.551866+0000","last_clean":"2023-10-21T05:26:03.551866+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:26:03.551866+0000","last_undegraded":"2023-10-21T05:26:03.551866+0000","last_fullsized":"2023-10-21T05:26:03.551866+0000","mapping_epoch":14,"log_start":"150'593561","ondisk_log_start":"150'593561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7680122880,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":595328,"num_read_kb":2352044,"num_write":593382,"num_write_kb":2373528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3775,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.300348+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:25:59.300348+0000","last_peered":"2023-10-21T05:25:59.300348+0000","last_clean":"2023-10-21T05:25:59.300348+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:25:59.300348+0000","last_undegraded":"2023-10-21T05:25:59.300348+0000","last_fullsized":"2023-10-21T05:25:59.300348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'630560","reported_seq":1244397,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.569628+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:26:02.569628+0000","last_peered":"2023-10-21T05:26:02.569628+0000","last_clean":"2023-10-21T05:26:02.569628+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:26:02.569628+0000","last_undegraded":"2023-10-21T05:26:02.569628+0000","last_fullsized":"2023-10-21T05:26:02.569628+0000","mapping_epoch":14,"log_start":"150'620554","ondisk_log_start":"150'620554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8021266432,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":618355,"num_read_kb":2443856,"num_write":619846,"num_write_kb":2479384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7502,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.298881+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:25:59.298881+0000","last_peered":"2023-10-21T05:25:59.298881+0000","last_clean":"2023-10-21T05:25:59.298881+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:25:59.298881+0000","last_undegraded":"2023-10-21T05:25:59.298881+0000","last_fullsized":"2023-10-21T05:25:59.298881+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'578099","reported_seq":1145826,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:03.543206+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:26:03.543206+0000","last_peered":"2023-10-21T05:26:03.543206+0000","last_clean":"2023-10-21T05:26:03.543206+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:26:03.543206+0000","last_undegraded":"2023-10-21T05:26:03.543206+0000","last_fullsized":"2023-10-21T05:26:03.543206+0000","mapping_epoch":14,"log_start":"150'568059","ondisk_log_start":"150'568059","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3420,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 492s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7386669075,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":571854,"num_read_kb":2260220,"num_write":570439,"num_write_kb":2281750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'7874","reported_seq":11446,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:25:59.712164+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:25:59.712164+0000","last_peered":"2023-10-21T05:25:59.712164+0000","last_clean":"2023-10-21T05:25:59.712164+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:25:59.712164+0000","last_undegraded":"2023-10-21T05:25:59.712164+0000","last_fullsized":"2023-10-21T05:25:59.712164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":331,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:27.922031+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:24:27.922031+0000","last_peered":"2023-10-21T05:24:27.922031+0000","last_clean":"2023-10-21T05:24:27.922031+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:24:27.922031+0000","last_undegraded":"2023-10-21T05:24:27.922031+0000","last_fullsized":"2023-10-21T05:24:27.922031+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30673903635,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2372927,"num_read_kb":9377248,"num_write":2370077,"num_write_kb":9480302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19414691840,"data_stored":39018508326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40111,"ondisk_log_size":40111,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63820,"internal_metadata":0},"log_size":48397,"ondisk_log_size":48397,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739407,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11349512,"kb_used_data":9480384,"kb_used_omap":2787,"kb_used_meta":1866332,"kb_avail":83022328,"statfs":{"total":96636764160,"available":85014863872,"internally_reserved":0,"allocated":9707913216,"data_stored":19508790123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2854187,"internal_metadata":1911124693},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:27 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.88400000000000001,"15min":1.01},"min":{"1min":0.24099999999999999,"5min":0.215,"15min":0.186},"max":{"1min":3.1070000000000002,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.88100000000000001,"15min":0.98799999999999999},"min":{"1min":0.26800000000000002,"5min":0.191,"15min":0.191},"max":{"1min":3.1280000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.34200000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.89800000000000002,"15min":0.90900000000000003},"min":{"1min":0.26700000000000002,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.7330000000000001,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":0.99199999999999999},"min":{"1min":0.29599999999999999,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":2.6989999999999998,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739406,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6689696,"kb_used_data":4657172,"kb_used_omap":1338,"kb_used_meta":2031173,"kb_avail":87682144,"statfs":{"total":96636764160,"available":89786515456,"internally_reserved":0,"allocated":4768944128,"data_stored":9579987775,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":2079922155},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.74299999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":2.6030000000000002,"5min":4.5010000000000003,"15min":6.0899999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.524,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.77500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.80800000000000005,"15min":0.88600000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":4.6289999999999996,"5min":4.6289999999999996,"15min":5.0339999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.90400000000000003,"15min":0.89300000000000002},"min":{"1min":0.222,"5min":0.22,"15min":0.22},"max":{"1min":3.621,"5min":5.1989999999999998,"15min":6.1230000000000002},"last":0.49199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5393692,"kb_used_data":4826324,"kb_used_omap":1499,"kb_used_meta":565860,"kb_avail":88978148,"statfs":{"total":96636764160,"available":91113623552,"internally_reserved":0,"allocated":4942155776,"data_stored":9931545622,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":579440831},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.78000000000000003},"min":{"1min":0.215,"5min":0.187,"15min":0.187},"max":{"1min":1.6060000000000001,"5min":5.923,"15min":5.923},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.71399999999999997,"15min":0.82499999999999996},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":1.1699999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.313}]},{"osd":2,"last update":"Sat Oct 21 05:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.1339999999999999,"15min":0.98199999999999998},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":4.867,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.0529999999999999,"15min":0.96899999999999997},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":3.173,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4940824576,"data_stored":9930870784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4767010816,"data_stored":9578872851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9706856448,"data_stored":19508764691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:26:05.078 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:26:05.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:26:05.614 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:26:05.634 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:26:14.479 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:26:14.474+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:26:25.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:26:26.149 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:26:26.153 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:26:26.175 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2753,"stamp":"2023-10-21T05:26:25.987026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30709606141,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2430783,"num_read_kb":9451786,"num_write":2379780,"num_write_kb":9516895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88703,"ondisk_log_size":88703,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23393352,"kb_used_data":19032560,"kb_used_omap":5625,"kb_used_meta":4355142,"kb_avail":259722168,"statfs":{"total":289910292480,"available":265955500032,"internally_reserved":0,"allocated":19489341440,"data_stored":39090651840,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5760129,"internal_metadata":4459666303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,4,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17948672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4251,"num_read_kb":16952,"num_write":4369,"num_write_kb":17476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001700"},"pg_stats":[{"pgid":"2.7","version":"151'7938","reported_seq":11586,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:22.617674+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:26:22.617674+0000","last_peered":"2023-10-21T05:26:22.617674+0000","last_clean":"2023-10-21T05:26:22.617674+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:26:22.617674+0000","last_undegraded":"2023-10-21T05:26:22.617674+0000","last_fullsized":"2023-10-21T05:26:22.617674+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7938,"log_dups_size":0,"ondisk_log_size":7938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'4061","reported_seq":7930,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:19.776784+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:26:19.776784+0000","last_peered":"2023-10-21T05:26:19.776784+0000","last_clean":"2023-10-21T05:26:19.776784+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:26:19.776784+0000","last_undegraded":"2023-10-21T05:26:19.776784+0000","last_fullsized":"2023-10-21T05:26:19.776784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4061,"log_dups_size":0,"ondisk_log_size":4061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'11848","reported_seq":17177,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:22.622959+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:26:22.622959+0000","last_peered":"2023-10-21T05:26:22.622959+0000","last_clean":"2023-10-21T05:26:22.622959+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:26:22.622959+0000","last_undegraded":"2023-10-21T05:26:22.622959+0000","last_fullsized":"2023-10-21T05:26:22.622959+0000","mapping_epoch":11,"log_start":"62'1800","ondisk_log_start":"62'1800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'7981","reported_seq":12644,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:22.611829+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:26:22.611829+0000","last_peered":"2023-10-21T05:26:22.611829+0000","last_clean":"2023-10-21T05:26:22.611829+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:26:22.611829+0000","last_undegraded":"2023-10-21T05:26:22.611829+0000","last_fullsized":"2023-10-21T05:26:22.611829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7981,"log_dups_size":0,"ondisk_log_size":7981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'598680","reported_seq":1183848,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:22.632860+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:26:22.632860+0000","last_peered":"2023-10-21T05:26:22.632860+0000","last_clean":"2023-10-21T05:26:22.632860+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:26:22.632860+0000","last_undegraded":"2023-10-21T05:26:22.632860+0000","last_fullsized":"2023-10-21T05:26:22.632860+0000","mapping_epoch":14,"log_start":"150'588662","ondisk_log_start":"150'588662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7594561536,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":589464,"num_read_kb":2329384,"num_write":588525,"num_write_kb":2354100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'16123","reported_seq":23366,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:19.807201+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:26:19.807201+0000","last_peered":"2023-10-21T05:26:19.807201+0000","last_clean":"2023-10-21T05:26:19.807201+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:26:19.807201+0000","last_undegraded":"2023-10-21T05:26:19.807201+0000","last_fullsized":"2023-10-21T05:26:19.807201+0000","mapping_epoch":11,"log_start":"99'6100","ondisk_log_start":"99'6100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'605710","reported_seq":1198878,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:23.650454+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:26:23.650454+0000","last_peered":"2023-10-21T05:26:23.650454+0000","last_clean":"2023-10-21T05:26:23.650454+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:26:23.650454+0000","last_undegraded":"2023-10-21T05:26:23.650454+0000","last_fullsized":"2023-10-21T05:26:23.650454+0000","mapping_epoch":14,"log_start":"150'595661","ondisk_log_start":"150'595661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7688671232,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":597504,"num_read_kb":2360720,"num_write":595469,"num_write_kb":2381876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3783,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:19.777585+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:26:19.777585+0000","last_peered":"2023-10-21T05:26:19.777585+0000","last_clean":"2023-10-21T05:26:19.777585+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:26:19.777585+0000","last_undegraded":"2023-10-21T05:26:19.777585+0000","last_fullsized":"2023-10-21T05:26:19.777585+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'632811","reported_seq":1248835,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:22.636342+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:26:22.636342+0000","last_peered":"2023-10-21T05:26:22.636342+0000","last_clean":"2023-10-21T05:26:22.636342+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:26:22.636342+0000","last_undegraded":"2023-10-21T05:26:22.636342+0000","last_fullsized":"2023-10-21T05:26:22.636342+0000","mapping_epoch":14,"log_start":"150'622754","ondisk_log_start":"150'622754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8030494720,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":620545,"num_read_kb":2452584,"num_write":622097,"num_write_kb":2488388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7519,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:19.777574+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:26:19.777574+0000","last_peered":"2023-10-21T05:26:19.777574+0000","last_clean":"2023-10-21T05:26:19.777574+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:26:19.777574+0000","last_undegraded":"2023-10-21T05:26:19.777574+0000","last_fullsized":"2023-10-21T05:26:19.777574+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'580270","reported_seq":1149964,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:23.664064+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:26:23.664064+0000","last_peered":"2023-10-21T05:26:23.664064+0000","last_clean":"2023-10-21T05:26:23.664064+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:26:23.664064+0000","last_undegraded":"2023-10-21T05:26:23.664064+0000","last_fullsized":"2023-10-21T05:26:23.664064+0000","mapping_epoch":14,"log_start":"150'570259","ondisk_log_start":"150'570259","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3515,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 512s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7395282963,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":573811,"num_read_kb":2268028,"num_write":572531,"num_write_kb":2290118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'7895","reported_seq":11475,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-21T05:26:22.621223+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:26:22.621223+0000","last_peered":"2023-10-21T05:26:22.621223+0000","last_clean":"2023-10-21T05:26:22.621223+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:26:22.621223+0000","last_undegraded":"2023-10-21T05:26:22.621223+0000","last_fullsized":"2023-10-21T05:26:22.621223+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":331,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-21T05:24:27.922031+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:24:27.922031+0000","last_peered":"2023-10-21T05:24:27.922031+0000","last_clean":"2023-10-21T05:24:27.922031+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:24:27.922031+0000","last_undegraded":"2023-10-21T05:24:27.922031+0000","last_fullsized":"2023-10-21T05:24:27.922031+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30709010451,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2381324,"num_read_kb":9410716,"num_write":2378622,"num_write_kb":9514482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19484758016,"data_stored":39088574502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40135,"ondisk_log_size":40135,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63820,"internal_metadata":0},"log_size":48469,"ondisk_log_size":48469,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739411,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11546376,"kb_used_data":9514816,"kb_used_omap":2787,"kb_used_meta":2028764,"kb_avail":82825464,"statfs":{"total":96636764160,"available":84813275136,"internally_reserved":0,"allocated":9743171584,"data_stored":19544048491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2854187,"internal_metadata":2077455061},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:27 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.88400000000000001,"15min":1.01},"min":{"1min":0.24099999999999999,"5min":0.215,"15min":0.186},"max":{"1min":3.1070000000000002,"5min":5.5609999999999999,"15min":27.184999999999999},"last":2.0910000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.88100000000000001,"15min":0.98799999999999999},"min":{"1min":0.26800000000000002,"5min":0.191,"15min":0.191},"max":{"1min":3.1280000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":1.089}]},{"osd":1,"last update":"Sat Oct 21 05:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.89800000000000002,"15min":0.90900000000000003},"min":{"1min":0.26700000000000002,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.7330000000000001,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":2.1549999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":0.99199999999999999},"min":{"1min":0.29599999999999999,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":2.6989999999999998,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":1.1419999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739410,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6363608,"kb_used_data":4673936,"kb_used_omap":1338,"kb_used_meta":1688325,"kb_avail":88008232,"statfs":{"total":96636764160,"available":90120429568,"internally_reserved":0,"allocated":4786110464,"data_stored":9597154111,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1728845803},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,0,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63700000000000001,"15min":0.71799999999999997},"min":{"1min":0.223,"5min":0.185,"15min":0.185},"max":{"1min":1.1759999999999999,"5min":4.5010000000000003,"15min":5.3920000000000003},"last":1.1759999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.27100000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.165,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.88100000000000001,"15min":0.89600000000000002},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.9950000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.012,"15min":0.89600000000000002},"min":{"1min":0.27600000000000002,"5min":0.222,"15min":0.22},"max":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.37}]}]},{"osd":0,"up_from":8,"seq":34359739410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5483368,"kb_used_data":4843808,"kb_used_omap":1499,"kb_used_meta":638052,"kb_avail":88888472,"statfs":{"total":96636764160,"available":91021795328,"internally_reserved":0,"allocated":4960059392,"data_stored":9949449238,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":653365439},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.78000000000000003},"min":{"1min":0.215,"5min":0.187,"15min":0.187},"max":{"1min":1.6060000000000001,"5min":5.923,"15min":5.923},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.71399999999999997,"15min":0.82499999999999996},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.20899999999999999},"max":{"1min":1.1699999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.71999999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.1339999999999999,"15min":0.98199999999999998},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":4.867,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.0529999999999999,"15min":0.96899999999999997},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":3.173,"5min":6.1779999999999999,"15min":6.1779999999999999},"last":1.157}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13104,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4958711808,"data_stored":9948758016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4784140288,"data_stored":9596002323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9741905920,"data_stored":19543814163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:26:26.181 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:26:26.181 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:26:44.411 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:26:44.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:26:44.411 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:26:44.408+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:26:44.412 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:26:44.408+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:26:46.183 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:26:46.732 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:26:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:26:46.752 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2763,"stamp":"2023-10-21T05:26:45.990331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30745003773,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2439150,"num_read_kb":9485142,"num_write":2388418,"num_write_kb":9551447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88828,"ondisk_log_size":88828,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23681104,"kb_used_data":19101236,"kb_used_omap":4357,"kb_used_meta":4575482,"kb_avail":259434416,"statfs":{"total":289910292480,"available":265660841984,"internally_reserved":0,"allocated":19559665664,"data_stored":39160964151,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4462201,"internal_metadata":4685293959},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2,7,9,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17805312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4141,"num_read_kb":16504,"num_write":4346,"num_write_kb":17384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002010"},"pg_stats":[{"pgid":"2.7","version":"152'7954","reported_seq":11611,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:43.482294+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:26:43.482294+0000","last_peered":"2023-10-21T05:26:43.482294+0000","last_clean":"2023-10-21T05:26:43.482294+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:26:43.482294+0000","last_undegraded":"2023-10-21T05:26:43.482294+0000","last_fullsized":"2023-10-21T05:26:43.482294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'4069","reported_seq":7943,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:41.916544+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:26:41.916544+0000","last_peered":"2023-10-21T05:26:41.916544+0000","last_clean":"2023-10-21T05:26:41.916544+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:26:41.916544+0000","last_undegraded":"2023-10-21T05:26:41.916544+0000","last_fullsized":"2023-10-21T05:26:41.916544+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'11872","reported_seq":17214,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:43.490441+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:26:43.490441+0000","last_peered":"2023-10-21T05:26:43.490441+0000","last_clean":"2023-10-21T05:26:43.490441+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:26:43.490441+0000","last_undegraded":"2023-10-21T05:26:43.490441+0000","last_fullsized":"2023-10-21T05:26:43.490441+0000","mapping_epoch":11,"log_start":"62'1800","ondisk_log_start":"62'1800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'7997","reported_seq":12669,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:43.482702+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:26:43.482702+0000","last_peered":"2023-10-21T05:26:43.482702+0000","last_clean":"2023-10-21T05:26:43.482702+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:26:43.482702+0000","last_undegraded":"2023-10-21T05:26:43.482702+0000","last_fullsized":"2023-10-21T05:26:43.482702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'600755","reported_seq":1187906,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:42.722575+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:26:42.722575+0000","last_peered":"2023-10-21T05:26:42.722575+0000","last_clean":"2023-10-21T05:26:42.722575+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:26:42.722575+0000","last_undegraded":"2023-10-21T05:26:42.722575+0000","last_fullsized":"2023-10-21T05:26:42.722575+0000","mapping_epoch":14,"log_start":"150'590662","ondisk_log_start":"150'590662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7603064832,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":591449,"num_read_kb":2337300,"num_write":590600,"num_write_kb":2362400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"152'16162","reported_seq":23422,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:41.992815+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:26:41.992815+0000","last_peered":"2023-10-21T05:26:41.992815+0000","last_clean":"2023-10-21T05:26:41.992815+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:26:41.992815+0000","last_undegraded":"2023-10-21T05:26:41.992815+0000","last_fullsized":"2023-10-21T05:26:41.992815+0000","mapping_epoch":11,"log_start":"99'6100","ondisk_log_start":"99'6100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'607872","reported_seq":1203190,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:43.803377+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:26:43.803377+0000","last_peered":"2023-10-21T05:26:43.803377+0000","last_clean":"2023-10-21T05:26:43.803377+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:26:43.803377+0000","last_undegraded":"2023-10-21T05:26:43.803377+0000","last_fullsized":"2023-10-21T05:26:43.803377+0000","mapping_epoch":14,"log_start":"150'597861","ondisk_log_start":"150'597861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7697522688,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":599652,"num_read_kb":2369272,"num_write":597631,"num_write_kb":2390524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3792,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:40.258611+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:26:40.258611+0000","last_peered":"2023-10-21T05:26:40.258611+0000","last_clean":"2023-10-21T05:26:40.258611+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:26:40.258611+0000","last_undegraded":"2023-10-21T05:26:40.258611+0000","last_fullsized":"2023-10-21T05:26:40.258611+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'635089","reported_seq":1253350,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:42.718783+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:26:42.718783+0000","last_peered":"2023-10-21T05:26:42.718783+0000","last_clean":"2023-10-21T05:26:42.718783+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:26:42.718783+0000","last_undegraded":"2023-10-21T05:26:42.718783+0000","last_fullsized":"2023-10-21T05:26:42.718783+0000","mapping_epoch":14,"log_start":"150'625054","ondisk_log_start":"150'625054","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8039829504,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":622779,"num_read_kb":2461488,"num_write":624375,"num_write_kb":2497500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7536,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:40.257145+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:26:40.257145+0000","last_peered":"2023-10-21T05:26:40.257145+0000","last_clean":"2023-10-21T05:26:40.257145+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:26:40.257145+0000","last_undegraded":"2023-10-21T05:26:40.257145+0000","last_fullsized":"2023-10-21T05:26:40.257145+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'582461","reported_seq":1154164,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:43.817120+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:26:43.817120+0000","last_peered":"2023-10-21T05:26:43.817120+0000","last_clean":"2023-10-21T05:26:43.817120+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:26:43.817120+0000","last_undegraded":"2023-10-21T05:26:43.817120+0000","last_fullsized":"2023-10-21T05:26:43.817120+0000","mapping_epoch":14,"log_start":"150'572459","ondisk_log_start":"150'572459","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3597,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 532s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7403991059,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":575811,"num_read_kb":2276012,"num_write":574654,"num_write_kb":2298610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'7911","reported_seq":11500,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:43.486977+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:26:43.486977+0000","last_peered":"2023-10-21T05:26:43.486977+0000","last_clean":"2023-10-21T05:26:43.486977+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:26:43.486977+0000","last_undegraded":"2023-10-21T05:26:43.486977+0000","last_fullsized":"2023-10-21T05:26:43.486977+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":333,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:24.604036+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:26:24.604036+0000","last_peered":"2023-10-21T05:26:24.604036+0000","last_clean":"2023-10-21T05:26:24.604036+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:26:24.604036+0000","last_undegraded":"2023-10-21T05:26:24.604036+0000","last_fullsized":"2023-10-21T05:26:24.604036+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30744408083,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2389691,"num_read_kb":9444072,"num_write":2387260,"num_write_kb":9549034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19555573760,"data_stored":39159390246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40141,"ondisk_log_size":40141,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":48588,"ondisk_log_size":48588,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739415,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11643164,"kb_used_data":9549008,"kb_used_omap":1519,"kb_used_meta":2092624,"kb_avail":82728676,"statfs":{"total":96636764160,"available":84714164224,"internally_reserved":0,"allocated":9778184192,"data_stored":19579057128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1556259,"internal_metadata":2142847197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.82199999999999995,"15min":1.0089999999999999},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.186},"max":{"1min":2.0910000000000002,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.876,"15min":0.98499999999999999},"min":{"1min":0.26700000000000002,"5min":0.191,"15min":0.191},"max":{"1min":1.5880000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":1.6359999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:26:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.84699999999999998,"15min":0.90400000000000003},"min":{"1min":0.23300000000000001,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.6339999999999999,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.84899999999999998,"15min":0.98899999999999999},"min":{"1min":0.24099999999999999,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":1.53,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":0.39200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739414,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6464936,"kb_used_data":4690976,"kb_used_omap":1338,"kb_used_meta":1772613,"kb_avail":87906904,"statfs":{"total":96636764160,"available":90016669696,"internally_reserved":0,"allocated":4803559424,"data_stored":9614599100,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1815156715},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63700000000000001,"15min":0.71799999999999997},"min":{"1min":0.223,"5min":0.185,"15min":0.185},"max":{"1min":1.1759999999999999,"5min":4.5010000000000003,"15min":5.3920000000000003},"last":2.2679999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.27100000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.165,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":2.1890000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.88100000000000001,"15min":0.89600000000000002},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.9950000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":2.2290000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.012,"15min":0.89600000000000002},"min":{"1min":0.27600000000000002,"5min":0.222,"15min":0.22},"max":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":1.3440000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5573004,"kb_used_data":4861252,"kb_used_omap":1499,"kb_used_meta":710244,"kb_avail":88798836,"statfs":{"total":96636764160,"available":90930008064,"internally_reserved":0,"allocated":4977922048,"data_stored":9967307923,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":727290047},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,4,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.80100000000000005,"15min":0.80200000000000005},"min":{"1min":0.28499999999999998,"5min":0.187,"15min":0.187},"max":{"1min":4.8090000000000002,"5min":5.923,"15min":5.923},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.80700000000000005,"15min":0.83399999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20899999999999999},"max":{"1min":4.8639999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.32800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:26:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.0649999999999999,"15min":0.96499999999999997},"min":{"1min":0.224,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":2.6059999999999999,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.375},{"interface":"front","average":{"1min":1.129,"5min":1.0509999999999999,"15min":0.98299999999999998},"min":{"1min":0.32600000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":3.9089999999999998,"5min":4.0129999999999999,"15min":6.1779999999999999},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4976545792,"data_stored":9966592000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4801675264,"data_stored":9613537299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9777352704,"data_stored":19579260947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:26:46.754 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:26:46.755 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:27:06.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:27:07.285 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:27:07.286 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:27:07.305 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2773,"stamp":"2023-10-21T05:27:05.993792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30780491517,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2447528,"num_read_kb":9518582,"num_write":2397058,"num_write_kb":9586007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88789,"ondisk_log_size":88789,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23204792,"kb_used_data":19172704,"kb_used_omap":4357,"kb_used_meta":4027706,"kb_avail":259910728,"statfs":{"total":289910292480,"available":266148585472,"internally_reserved":0,"allocated":19632848896,"data_stored":39234147383,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4462201,"internal_metadata":4124371335},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,9,11,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17948672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4200,"num_read_kb":16760,"num_write":4352,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002127"},"pg_stats":[{"pgid":"2.7","version":"152'7974","reported_seq":11640,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:00.736969+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:27:00.736969+0000","last_peered":"2023-10-21T05:27:00.736969+0000","last_clean":"2023-10-21T05:27:00.736969+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:27:00.736969+0000","last_undegraded":"2023-10-21T05:27:00.736969+0000","last_fullsized":"2023-10-21T05:27:00.736969+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'4080","reported_seq":7958,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:59.159848+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:26:59.159848+0000","last_peered":"2023-10-21T05:26:59.159848+0000","last_clean":"2023-10-21T05:26:59.159848+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:26:59.159848+0000","last_undegraded":"2023-10-21T05:26:59.159848+0000","last_fullsized":"2023-10-21T05:26:59.159848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'11903","reported_seq":17257,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:00.736915+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:27:00.736915+0000","last_peered":"2023-10-21T05:27:00.736915+0000","last_clean":"2023-10-21T05:27:00.736915+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:27:00.736915+0000","last_undegraded":"2023-10-21T05:27:00.736915+0000","last_fullsized":"2023-10-21T05:27:00.736915+0000","mapping_epoch":11,"log_start":"64'1900","ondisk_log_start":"64'1900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'8018","reported_seq":12698,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:00.736754+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:27:00.736754+0000","last_peered":"2023-10-21T05:27:00.736754+0000","last_clean":"2023-10-21T05:27:00.736754+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:27:00.736754+0000","last_undegraded":"2023-10-21T05:27:00.736754+0000","last_fullsized":"2023-10-21T05:27:00.736754+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'602829","reported_seq":1192041,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:02.843990+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:27:02.843990+0000","last_peered":"2023-10-21T05:27:02.843990+0000","last_clean":"2023-10-21T05:27:02.843990+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:27:02.843990+0000","last_undegraded":"2023-10-21T05:27:02.843990+0000","last_fullsized":"2023-10-21T05:27:02.843990+0000","mapping_epoch":14,"log_start":"151'592762","ondisk_log_start":"151'592762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7611551744,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":593510,"num_read_kb":2345532,"num_write":592674,"num_write_kb":2370696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"152'16211","reported_seq":23487,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:02.595184+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:27:02.595184+0000","last_peered":"2023-10-21T05:27:02.595184+0000","last_clean":"2023-10-21T05:27:02.595184+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:27:02.595184+0000","last_undegraded":"2023-10-21T05:27:02.595184+0000","last_fullsized":"2023-10-21T05:27:02.595184+0000","mapping_epoch":11,"log_start":"99'6200","ondisk_log_start":"99'6200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'610064","reported_seq":1207489,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:03.964691+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:27:03.964691+0000","last_peered":"2023-10-21T05:27:03.964691+0000","last_clean":"2023-10-21T05:27:03.964691+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:27:03.964691+0000","last_undegraded":"2023-10-21T05:27:03.964691+0000","last_fullsized":"2023-10-21T05:27:03.964691+0000","mapping_epoch":14,"log_start":"151'600061","ondisk_log_start":"151'600061","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7706529792,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":601762,"num_read_kb":2377692,"num_write":599823,"num_write_kb":2399292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3800,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:00.738310+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:27:00.738310+0000","last_peered":"2023-10-21T05:27:00.738310+0000","last_clean":"2023-10-21T05:27:00.738310+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:27:00.738310+0000","last_undegraded":"2023-10-21T05:27:00.738310+0000","last_fullsized":"2023-10-21T05:27:00.738310+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'637384","reported_seq":1257818,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:02.826290+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:27:02.826290+0000","last_peered":"2023-10-21T05:27:02.826290+0000","last_clean":"2023-10-21T05:27:02.826290+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:27:02.826290+0000","last_undegraded":"2023-10-21T05:27:02.826290+0000","last_fullsized":"2023-10-21T05:27:02.826290+0000","mapping_epoch":14,"log_start":"151'627354","ondisk_log_start":"151'627354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8049270784,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":624953,"num_read_kb":2470156,"num_write":626670,"num_write_kb":2506680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7552,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:00.737827+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:27:00.737827+0000","last_peered":"2023-10-21T05:27:00.737827+0000","last_clean":"2023-10-21T05:27:00.737827+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:27:00.737827+0000","last_undegraded":"2023-10-21T05:27:00.737827+0000","last_fullsized":"2023-10-21T05:27:00.737827+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'584609","reported_seq":1158360,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:03.952234+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:27:03.952234+0000","last_peered":"2023-10-21T05:27:03.952234+0000","last_clean":"2023-10-21T05:27:03.952234+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:27:03.952234+0000","last_undegraded":"2023-10-21T05:27:03.952234+0000","last_fullsized":"2023-10-21T05:27:03.952234+0000","mapping_epoch":14,"log_start":"151'574559","ondisk_log_start":"151'574559","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3683,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 552s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7412543507,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":577844,"num_read_kb":2284132,"num_write":576733,"num_write_kb":2306926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'7931","reported_seq":11528,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:00.736957+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:27:00.736957+0000","last_peered":"2023-10-21T05:27:00.736957+0000","last_clean":"2023-10-21T05:27:00.736957+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:27:00.736957+0000","last_undegraded":"2023-10-21T05:27:00.736957+0000","last_fullsized":"2023-10-21T05:27:00.736957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":333,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:24.604036+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:26:24.604036+0000","last_peered":"2023-10-21T05:26:24.604036+0000","last_clean":"2023-10-21T05:26:24.604036+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:26:24.604036+0000","last_undegraded":"2023-10-21T05:26:24.604036+0000","last_fullsized":"2023-10-21T05:26:24.604036+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30779895827,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2398069,"num_read_kb":9477512,"num_write":2395900,"num_write_kb":9583594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19629056000,"data_stored":39232872486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40150,"ondisk_log_size":40150,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":48540,"ondisk_log_size":48540,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739420,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10984088,"kb_used_data":9586256,"kb_used_omap":1519,"kb_used_meta":1396304,"kb_avail":83387752,"statfs":{"total":96636764160,"available":85389058048,"internally_reserved":0,"allocated":9816326144,"data_stored":19617199080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1556259,"internal_metadata":1429815517},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.82199999999999995,"15min":1.0089999999999999},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.186},"max":{"1min":2.0910000000000002,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.876,"15min":0.98499999999999999},"min":{"1min":0.26700000000000002,"5min":0.191,"15min":0.191},"max":{"1min":1.5880000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Sat Oct 21 05:26:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.84699999999999998,"15min":0.90400000000000003},"min":{"1min":0.23300000000000001,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.6339999999999999,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.254},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.84899999999999998,"15min":0.98899999999999999},"min":{"1min":0.24099999999999999,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":1.53,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":1.004}]}]},{"osd":1,"up_from":8,"seq":34359739418,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6554044,"kb_used_data":4707892,"kb_used_omap":1338,"kb_used_meta":1844805,"kb_avail":87817796,"statfs":{"total":96636764160,"available":89925423104,"internally_reserved":0,"allocated":4820881408,"data_stored":9631921084,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1889081323},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63700000000000001,"15min":0.71799999999999997},"min":{"1min":0.223,"5min":0.185,"15min":0.185},"max":{"1min":1.1759999999999999,"5min":4.5010000000000003,"15min":5.3920000000000003},"last":2.0910000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.27100000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.165,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":1.002}]},{"osd":2,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.88100000000000001,"15min":0.89600000000000002},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.9950000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.012,"15min":0.89600000000000002},"min":{"1min":0.27600000000000002,"5min":0.222,"15min":0.22},"max":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":1.889}]}]},{"osd":0,"up_from":8,"seq":34359739418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5666660,"kb_used_data":4878556,"kb_used_omap":1499,"kb_used_meta":786596,"kb_avail":88705180,"statfs":{"total":96636764160,"available":90834104320,"internally_reserved":0,"allocated":4995641344,"data_stored":9985027219,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":805474495},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.80100000000000005,"15min":0.80200000000000005},"min":{"1min":0.28499999999999998,"5min":0.187,"15min":0.187},"max":{"1min":4.8090000000000002,"5min":5.923,"15min":5.923},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.80700000000000005,"15min":0.83399999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20899999999999999},"max":{"1min":4.8639999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:26:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.0649999999999999,"15min":0.96499999999999997},"min":{"1min":0.224,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":2.6059999999999999,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.129,"5min":1.0509999999999999,"15min":0.98299999999999998},"min":{"1min":0.32600000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":3.9089999999999998,"5min":4.0129999999999999,"15min":6.1779999999999999},"last":1.95}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4994437120,"data_stored":9984483328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4819173376,"data_stored":9631035411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9815445504,"data_stored":19617353747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:27:07.307 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:27:07.308 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:27:13.588 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:13.584+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:27:18.539 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.532+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:27:18.540 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:27:18.540 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:27:18.540 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:27:18.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:27:18.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:27:18.541 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:27:18.542 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:27:18.542 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:18.537+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:27:27.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:27:27.848 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:27:27.849 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:27:27.870 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2783,"stamp":"2023-10-21T05:27:25.997110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30815266557,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2455959,"num_read_kb":9552246,"num_write":2405546,"num_write_kb":9619959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88973,"ondisk_log_size":88973,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23524420,"kb_used_data":19240428,"kb_used_omap":4357,"kb_used_meta":4279610,"kb_avail":259591100,"statfs":{"total":289910292480,"available":265821286400,"internally_reserved":0,"allocated":19702198272,"data_stored":39303496759,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4462201,"internal_metadata":4382321031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1,9,9,6,9,16],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17317888,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4258,"num_read_kb":17004,"num_write":4231,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001919"},"pg_stats":[{"pgid":"2.7","version":"152'8000","reported_seq":11674,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:21.218267+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:27:21.218267+0000","last_peered":"2023-10-21T05:27:21.218267+0000","last_clean":"2023-10-21T05:27:21.218267+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:27:21.218267+0000","last_undegraded":"2023-10-21T05:27:21.218267+0000","last_fullsized":"2023-10-21T05:27:21.218267+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'4094","reported_seq":7976,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:20.097550+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:27:20.097550+0000","last_peered":"2023-10-21T05:27:20.097550+0000","last_clean":"2023-10-21T05:27:20.097550+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:27:20.097550+0000","last_undegraded":"2023-10-21T05:27:20.097550+0000","last_fullsized":"2023-10-21T05:27:20.097550+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4094,"log_dups_size":0,"ondisk_log_size":4094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'11942","reported_seq":17308,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:21.218462+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:27:21.218462+0000","last_peered":"2023-10-21T05:27:21.218462+0000","last_clean":"2023-10-21T05:27:21.218462+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:27:21.218462+0000","last_undegraded":"2023-10-21T05:27:21.218462+0000","last_fullsized":"2023-10-21T05:27:21.218462+0000","mapping_epoch":11,"log_start":"64'1900","ondisk_log_start":"64'1900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'8045","reported_seq":12733,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:21.220237+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:27:21.220237+0000","last_peered":"2023-10-21T05:27:21.220237+0000","last_clean":"2023-10-21T05:27:21.220237+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:27:21.220237+0000","last_undegraded":"2023-10-21T05:27:21.220237+0000","last_fullsized":"2023-10-21T05:27:21.220237+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8045,"log_dups_size":0,"ondisk_log_size":8045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'604932","reported_seq":1196224,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:22.940501+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:27:22.940501+0000","last_peered":"2023-10-21T05:27:22.940501+0000","last_clean":"2023-10-21T05:27:22.940501+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:27:22.940501+0000","last_undegraded":"2023-10-21T05:27:22.940501+0000","last_fullsized":"2023-10-21T05:27:22.940501+0000","mapping_epoch":14,"log_start":"151'594862","ondisk_log_start":"151'594862","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7620186112,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":595588,"num_read_kb":2353824,"num_write":594777,"num_write_kb":2379108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"152'16268","reported_seq":23560,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:22.309076+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:27:22.309076+0000","last_peered":"2023-10-21T05:27:22.309076+0000","last_clean":"2023-10-21T05:27:22.309076+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:27:22.309076+0000","last_undegraded":"2023-10-21T05:27:22.309076+0000","last_fullsized":"2023-10-21T05:27:22.309076+0000","mapping_epoch":11,"log_start":"99'6200","ondisk_log_start":"99'6200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'612201","reported_seq":1211738,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:24.043028+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:27:24.043028+0000","last_peered":"2023-10-21T05:27:24.043028+0000","last_clean":"2023-10-21T05:27:24.043028+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:27:24.043028+0000","last_undegraded":"2023-10-21T05:27:24.043028+0000","last_fullsized":"2023-10-21T05:27:24.043028+0000","mapping_epoch":14,"log_start":"151'602161","ondisk_log_start":"151'602161","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7715262464,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":603874,"num_read_kb":2386136,"num_write":601960,"num_write_kb":2407840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3808,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:21.233281+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:27:21.233281+0000","last_peered":"2023-10-21T05:27:21.233281+0000","last_clean":"2023-10-21T05:27:21.233281+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:27:21.233281+0000","last_undegraded":"2023-10-21T05:27:21.233281+0000","last_fullsized":"2023-10-21T05:27:21.233281+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'639557","reported_seq":1262192,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:22.938649+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:27:22.938649+0000","last_peered":"2023-10-21T05:27:22.938649+0000","last_clean":"2023-10-21T05:27:22.938649+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:27:22.938649+0000","last_undegraded":"2023-10-21T05:27:22.938649+0000","last_fullsized":"2023-10-21T05:27:22.938649+0000","mapping_epoch":14,"log_start":"151'629554","ondisk_log_start":"151'629554","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8058179584,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":627152,"num_read_kb":2478928,"num_write":628843,"num_write_kb":2515372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7568,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:21.219746+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:27:21.219746+0000","last_peered":"2023-10-21T05:27:21.219746+0000","last_clean":"2023-10-21T05:27:21.219746+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:27:21.219746+0000","last_undegraded":"2023-10-21T05:27:21.219746+0000","last_fullsized":"2023-10-21T05:27:21.219746+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'586791","reported_seq":1162592,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:24.043965+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:27:24.043965+0000","last_peered":"2023-10-21T05:27:24.043965+0000","last_clean":"2023-10-21T05:27:24.043965+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:27:24.043965+0000","last_undegraded":"2023-10-21T05:27:24.043965+0000","last_fullsized":"2023-10-21T05:27:24.043965+0000","mapping_epoch":14,"log_start":"151'576759","ondisk_log_start":"151'576759","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3814,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 572s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7421042707,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":579886,"num_read_kb":2292288,"num_write":578808,"num_write_kb":2315226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'7957","reported_seq":11562,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:27:21.218885+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:27:21.218885+0000","last_peered":"2023-10-21T05:27:21.218885+0000","last_clean":"2023-10-21T05:27:21.218885+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:27:21.218885+0000","last_undegraded":"2023-10-21T05:27:21.218885+0000","last_fullsized":"2023-10-21T05:27:21.218885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":333,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:24.604036+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:26:24.604036+0000","last_peered":"2023-10-21T05:26:24.604036+0000","last_clean":"2023-10-21T05:26:24.604036+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:26:24.604036+0000","last_undegraded":"2023-10-21T05:26:24.604036+0000","last_fullsized":"2023-10-21T05:26:24.604036+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30814670867,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2406500,"num_read_kb":9511176,"num_write":2404388,"num_write_kb":9617546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19698421760,"data_stored":39302238246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40145,"ondisk_log_size":40145,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":48729,"ondisk_log_size":48729,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739424,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11180128,"kb_used_data":9619992,"kb_used_omap":1519,"kb_used_meta":1558608,"kb_avail":83191712,"statfs":{"total":96636764160,"available":85188313088,"internally_reserved":0,"allocated":9850871808,"data_stored":19651744744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1556259,"internal_metadata":1596014813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.82199999999999995,"15min":1.0089999999999999},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.186},"max":{"1min":2.0910000000000002,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.38},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.876,"15min":0.98499999999999999},"min":{"1min":0.26700000000000002,"5min":0.191,"15min":0.191},"max":{"1min":1.5880000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:26:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.84699999999999998,"15min":0.90400000000000003},"min":{"1min":0.23300000000000001,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.6339999999999999,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.84899999999999998,"15min":0.98899999999999999},"min":{"1min":0.24099999999999999,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":1.53,"5min":5.7699999999999996,"15min":9.0020000000000007},"last":0.57599999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739422,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6668996,"kb_used_data":4724796,"kb_used_omap":1338,"kb_used_meta":1942853,"kb_avail":87702844,"statfs":{"total":96636764160,"available":89807712256,"internally_reserved":0,"allocated":4838191104,"data_stored":9649230780,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":1989482475},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,3,6,9,16],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63700000000000001,"15min":0.71799999999999997},"min":{"1min":0.223,"5min":0.185,"15min":0.185},"max":{"1min":1.1759999999999999,"5min":4.5010000000000003,"15min":5.3920000000000003},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.27100000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.165,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.218}]},{"osd":2,"last update":"Sat Oct 21 05:26:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.88100000000000001,"15min":0.89600000000000002},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.9950000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.012,"15min":0.89600000000000002},"min":{"1min":0.27600000000000002,"5min":0.222,"15min":0.22},"max":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.28699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5675296,"kb_used_data":4895640,"kb_used_omap":1499,"kb_used_meta":778148,"kb_avail":88696544,"statfs":{"total":96636764160,"available":90825261056,"internally_reserved":0,"allocated":5013135360,"data_stored":10002521235,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":796823743},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.80100000000000005,"15min":0.80200000000000005},"min":{"1min":0.28499999999999998,"5min":0.187,"15min":0.187},"max":{"1min":4.8090000000000002,"5min":5.923,"15min":5.923},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.80700000000000005,"15min":0.83399999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20899999999999999},"max":{"1min":4.8639999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:26:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.0649999999999999,"15min":0.96499999999999997},"min":{"1min":0.224,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":2.6059999999999999,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.129,"5min":1.0509999999999999,"15min":0.98299999999999998},"min":{"1min":0.32600000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":3.9089999999999998,"5min":4.0129999999999999,"15min":6.1779999999999999},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5011963904,"data_stored":10002010112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4836421632,"data_stored":9648283667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9850036224,"data_stored":19651944467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:27:27.872 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:27:27.872 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:27:30.870 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.868+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:27:30.871 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.868+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:27:30.884 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:27:30.885 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:27:30.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:27:30.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:27:30.886 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:27:30.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:27:30.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:27:30.887 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:27:30.883+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:27:32.512 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:27:32.511+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:27:47.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:27:48.397 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:27:48.398 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:27:48.419 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2795,"stamp":"2023-10-21T05:27:48.000915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30850242301,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2464391,"num_read_kb":9585894,"num_write":2414057,"num_write_kb":9654003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89052,"ondisk_log_size":89052,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23903240,"kb_used_data":19307952,"kb_used_omap":4357,"kb_used_meta":4590906,"kb_avail":259212280,"statfs":{"total":289910292480,"available":265433374720,"internally_reserved":0,"allocated":19771342848,"data_stored":39372629293,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4462201,"internal_metadata":4701088135},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,9,4,13,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17571840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4150,"num_read_kb":16564,"num_write":4276,"num_write_kb":17104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001664"},"pg_stats":[{"pgid":"2.7","version":"153'8022","reported_seq":11705,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:44.110194+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:27:44.110194+0000","last_peered":"2023-10-21T05:27:44.110194+0000","last_clean":"2023-10-21T05:27:44.110194+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:27:44.110194+0000","last_undegraded":"2023-10-21T05:27:44.110194+0000","last_fullsized":"2023-10-21T05:27:44.110194+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'4105","reported_seq":7993,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:44.096329+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:27:44.096329+0000","last_peered":"2023-10-21T05:27:44.096329+0000","last_clean":"2023-10-21T05:27:44.096329+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:27:44.096329+0000","last_undegraded":"2023-10-21T05:27:44.096329+0000","last_fullsized":"2023-10-21T05:27:44.096329+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'11975","reported_seq":17354,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:44.122594+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:27:44.122594+0000","last_peered":"2023-10-21T05:27:44.122594+0000","last_clean":"2023-10-21T05:27:44.122594+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:27:44.122594+0000","last_undegraded":"2023-10-21T05:27:44.122594+0000","last_fullsized":"2023-10-21T05:27:44.122594+0000","mapping_epoch":11,"log_start":"64'1900","ondisk_log_start":"64'1900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'8067","reported_seq":12764,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:44.110363+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:27:44.110363+0000","last_peered":"2023-10-21T05:27:44.110363+0000","last_clean":"2023-10-21T05:27:44.110363+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:27:44.110363+0000","last_undegraded":"2023-10-21T05:27:44.110363+0000","last_fullsized":"2023-10-21T05:27:44.110363+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'607069","reported_seq":1200418,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:43.047421+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:27:43.047421+0000","last_peered":"2023-10-21T05:27:43.047421+0000","last_clean":"2023-10-21T05:27:43.047421+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:27:43.047421+0000","last_undegraded":"2023-10-21T05:27:43.047421+0000","last_fullsized":"2023-10-21T05:27:43.047421+0000","mapping_epoch":14,"log_start":"151'597062","ondisk_log_start":"151'597062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7628980224,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":597646,"num_read_kb":2362040,"num_write":596914,"num_write_kb":2387656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'16328","reported_seq":23637,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:42.645256+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:27:42.645256+0000","last_peered":"2023-10-21T05:27:42.645256+0000","last_clean":"2023-10-21T05:27:42.645256+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:27:42.645256+0000","last_undegraded":"2023-10-21T05:27:42.645256+0000","last_fullsized":"2023-10-21T05:27:42.645256+0000","mapping_epoch":11,"log_start":"101'6300","ondisk_log_start":"101'6300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'614351","reported_seq":1216002,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:44.137627+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:27:44.137627+0000","last_peered":"2023-10-21T05:27:44.137627+0000","last_clean":"2023-10-21T05:27:44.137627+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:27:44.137627+0000","last_undegraded":"2023-10-21T05:27:44.137627+0000","last_fullsized":"2023-10-21T05:27:44.137627+0000","mapping_epoch":14,"log_start":"151'604261","ondisk_log_start":"151'604261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7724093440,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":605992,"num_read_kb":2394572,"num_write":604110,"num_write_kb":2416440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3818,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:41.696964+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:27:41.696964+0000","last_peered":"2023-10-21T05:27:41.696964+0000","last_clean":"2023-10-21T05:27:41.696964+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:27:41.696964+0000","last_undegraded":"2023-10-21T05:27:41.696964+0000","last_fullsized":"2023-10-21T05:27:41.696964+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'641771","reported_seq":1266600,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:43.023581+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:27:43.023581+0000","last_peered":"2023-10-21T05:27:43.023581+0000","last_clean":"2023-10-21T05:27:43.023581+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:27:43.023581+0000","last_undegraded":"2023-10-21T05:27:43.023581+0000","last_fullsized":"2023-10-21T05:27:43.023581+0000","mapping_epoch":14,"log_start":"151'631754","ondisk_log_start":"151'631754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8067284992,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":629347,"num_read_kb":2487696,"num_write":631057,"num_write_kb":2524228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7585,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:41.711389+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:27:41.711389+0000","last_peered":"2023-10-21T05:27:41.711389+0000","last_clean":"2023-10-21T05:27:41.711389+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:27:41.711389+0000","last_undegraded":"2023-10-21T05:27:41.711389+0000","last_fullsized":"2023-10-21T05:27:41.711389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'588899","reported_seq":1166775,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:44.139583+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:27:44.139583+0000","last_peered":"2023-10-21T05:27:44.139583+0000","last_clean":"2023-10-21T05:27:44.139583+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:27:44.139583+0000","last_undegraded":"2023-10-21T05:27:44.139583+0000","last_fullsized":"2023-10-21T05:27:44.139583+0000","mapping_epoch":14,"log_start":"151'578859","ondisk_log_start":"151'578859","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":3937,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 592s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7429287955,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":581947,"num_read_kb":2300516,"num_write":580818,"num_write_kb":2323266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'7979","reported_seq":11593,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:44.116885+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:27:44.116885+0000","last_peered":"2023-10-21T05:27:44.116885+0000","last_clean":"2023-10-21T05:27:44.116885+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:27:44.116885+0000","last_undegraded":"2023-10-21T05:27:44.116885+0000","last_fullsized":"2023-10-21T05:27:44.116885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":7979,"log_dups_size":0,"ondisk_log_size":7979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":334,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:28.714743+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:26:28.714743+0000","last_peered":"2023-10-21T05:26:28.714743+0000","last_clean":"2023-10-21T05:26:28.714743+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:26:28.714743+0000","last_undegraded":"2023-10-21T05:26:28.714743+0000","last_fullsized":"2023-10-21T05:26:28.714743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30849646611,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2414932,"num_read_kb":9544824,"num_write":2412899,"num_write_kb":9651590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19768102912,"data_stored":39371919398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40154,"ondisk_log_size":40154,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":48799,"ondisk_log_size":48799,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739428,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11401164,"kb_used_data":9653828,"kb_used_omap":1519,"kb_used_meta":1745808,"kb_avail":82970676,"statfs":{"total":96636764160,"available":84961972224,"internally_reserved":0,"allocated":9885519872,"data_stored":19686388794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1556259,"internal_metadata":1787707613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:31 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.83299999999999996,"15min":0.997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":4.0309999999999997,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.86099999999999999,"15min":0.997},"min":{"1min":0.26600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":2.468,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.97999999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:27:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.84899999999999998,"15min":0.88400000000000001},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":4.0810000000000004,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":1,"5min":0.83199999999999996,"15min":0.95599999999999996},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":4.1159999999999997,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.52900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739426,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6757644,"kb_used_data":4741252,"kb_used_omap":1338,"kb_used_meta":2015045,"kb_avail":87614196,"statfs":{"total":96636764160,"available":89716936704,"internally_reserved":0,"allocated":4855042048,"data_stored":9666077710,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1370133,"internal_metadata":2063407083},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,4,1,9,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:24 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.72499999999999998,"15min":0.75700000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":3.665,"5min":4.5010000000000003,"15min":5.3920000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.4379999999999999,"5min":0.90100000000000002,"15min":0.82299999999999995},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.968,"5min":6.968,"15min":6.968},"last":0.503}]},{"osd":2,"last update":"Sat Oct 21 05:27:24 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.014,"15min":0.91200000000000003},"min":{"1min":0.26000000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":3.339,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.024,"15min":0.92100000000000004},"min":{"1min":0.25,"5min":0.222,"15min":0.22},"max":{"1min":2.5419999999999998,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.93799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359739426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5744432,"kb_used_data":4912872,"kb_used_omap":1499,"kb_used_meta":830052,"kb_avail":88627408,"statfs":{"total":96636764160,"available":90754465792,"internally_reserved":0,"allocated":5030780928,"data_stored":10020162789,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":849973439},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:27:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.752,"15min":0.76800000000000002},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.3089999999999999,"5min":5.923,"15min":5.923},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.77200000000000002,"15min":0.80900000000000005},"min":{"1min":0.28799999999999998,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":1.5669999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":1.655}]},{"osd":2,"last update":"Sat Oct 21 05:27:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1240000000000001,"15min":0.97099999999999997},"min":{"1min":0.254,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":4.6120000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"last":0.378},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.98099999999999998,"15min":0.95799999999999996},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.95,"5min":3.9089999999999998,"15min":6.1779999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5029769216,"data_stored":10019815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4853460992,"data_stored":9665323027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9884872704,"data_stored":19686780947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:27:48.420 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:27:48.421 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:28:08.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:28:08.964 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:28:08.965 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:28:08.987 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2805,"stamp":"2023-10-21T05:28:08.004317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30884722429,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2472722,"num_read_kb":9619190,"num_write":2422473,"num_write_kb":9687667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89267,"ondisk_log_size":89267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23532952,"kb_used_data":19375676,"kb_used_omap":4359,"kb_used_meta":4152888,"kb_avail":259582568,"statfs":{"total":289910292480,"available":265812549632,"internally_reserved":0,"allocated":19840692224,"data_stored":39441978669,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4463880,"internal_metadata":4252558072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,3,6,16,13,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17424384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4251,"num_read_kb":16992,"num_write":4253,"num_write_kb":17012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002025"},"pg_stats":[{"pgid":"2.7","version":"153'8054","reported_seq":11745,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.177197+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:28:02.177197+0000","last_peered":"2023-10-21T05:28:02.177197+0000","last_clean":"2023-10-21T05:28:02.177197+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:28:02.177197+0000","last_undegraded":"2023-10-21T05:28:02.177197+0000","last_fullsized":"2023-10-21T05:28:02.177197+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'4121","reported_seq":8013,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.103155+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:28:02.103155+0000","last_peered":"2023-10-21T05:28:02.103155+0000","last_clean":"2023-10-21T05:28:02.103155+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:28:02.103155+0000","last_undegraded":"2023-10-21T05:28:02.103155+0000","last_fullsized":"2023-10-21T05:28:02.103155+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'12023","reported_seq":17415,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.177112+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:28:02.177112+0000","last_peered":"2023-10-21T05:28:02.177112+0000","last_clean":"2023-10-21T05:28:02.177112+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:28:02.177112+0000","last_undegraded":"2023-10-21T05:28:02.177112+0000","last_fullsized":"2023-10-21T05:28:02.177112+0000","mapping_epoch":11,"log_start":"66'2000","ondisk_log_start":"66'2000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'8099","reported_seq":12805,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.177208+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:28:02.177208+0000","last_peered":"2023-10-21T05:28:02.177208+0000","last_clean":"2023-10-21T05:28:02.177208+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:28:02.177208+0000","last_undegraded":"2023-10-21T05:28:02.177208+0000","last_fullsized":"2023-10-21T05:28:02.177208+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'609149","reported_seq":1204528,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:03.154881+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:28:03.154881+0000","last_peered":"2023-10-21T05:28:03.154881+0000","last_clean":"2023-10-21T05:28:03.154881+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:28:03.154881+0000","last_undegraded":"2023-10-21T05:28:03.154881+0000","last_fullsized":"2023-10-21T05:28:03.154881+0000","mapping_epoch":14,"log_start":"152'599062","ondisk_log_start":"152'599062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7637516288,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":599675,"num_read_kb":2370156,"num_write":598994,"num_write_kb":2395976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'16384","reported_seq":23710,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.177138+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:28:02.177138+0000","last_peered":"2023-10-21T05:28:02.177138+0000","last_clean":"2023-10-21T05:28:02.177138+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:28:02.177138+0000","last_undegraded":"2023-10-21T05:28:02.177138+0000","last_fullsized":"2023-10-21T05:28:02.177138+0000","mapping_epoch":11,"log_start":"101'6300","ondisk_log_start":"101'6300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'616409","reported_seq":1220164,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:04.282557+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:28:04.282557+0000","last_peered":"2023-10-21T05:28:04.282557+0000","last_clean":"2023-10-21T05:28:04.282557+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:28:04.282557+0000","last_undegraded":"2023-10-21T05:28:04.282557+0000","last_fullsized":"2023-10-21T05:28:04.282557+0000","mapping_epoch":14,"log_start":"152'606361","ondisk_log_start":"152'606361","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7732494336,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":608089,"num_read_kb":2402956,"num_write":606168,"num_write_kb":2424672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"134'170","reported_seq":3826,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.182406+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:28:02.182406+0000","last_peered":"2023-10-21T05:28:02.182406+0000","last_clean":"2023-10-21T05:28:02.182406+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:28:02.182406+0000","last_undegraded":"2023-10-21T05:28:02.182406+0000","last_fullsized":"2023-10-21T05:28:02.182406+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'643996","reported_seq":1271012,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:03.152452+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:28:03.152452+0000","last_peered":"2023-10-21T05:28:03.152452+0000","last_clean":"2023-10-21T05:28:03.152452+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:28:03.152452+0000","last_undegraded":"2023-10-21T05:28:03.152452+0000","last_fullsized":"2023-10-21T05:28:03.152452+0000","mapping_epoch":14,"log_start":"152'633954","ondisk_log_start":"152'633954","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8076406784,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":631532,"num_read_kb":2496416,"num_write":633282,"num_write_kb":2533128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"134'353","reported_seq":7601,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.183106+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:28:02.183106+0000","last_peered":"2023-10-21T05:28:02.183106+0000","last_clean":"2023-10-21T05:28:02.183106+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:28:02.183106+0000","last_undegraded":"2023-10-21T05:28:02.183106+0000","last_fullsized":"2023-10-21T05:28:02.183106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'591035","reported_seq":1170939,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:04.291728+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:28:04.291728+0000","last_peered":"2023-10-21T05:28:04.291728+0000","last_clean":"2023-10-21T05:28:04.291728+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:28:04.291728+0000","last_undegraded":"2023-10-21T05:28:04.291728+0000","last_fullsized":"2023-10-21T05:28:04.291728+0000","mapping_epoch":14,"log_start":"152'580959","ondisk_log_start":"152'580959","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4039,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 612s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7437709331,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":583967,"num_read_kb":2308592,"num_write":582871,"num_write_kb":2331478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'8011","reported_seq":11634,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:02.177091+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:28:02.177091+0000","last_peered":"2023-10-21T05:28:02.177091+0000","last_clean":"2023-10-21T05:28:02.177091+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:28:02.177091+0000","last_undegraded":"2023-10-21T05:28:02.177091+0000","last_fullsized":"2023-10-21T05:28:02.177091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":334,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-21T05:26:28.714743+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:26:28.714743+0000","last_peered":"2023-10-21T05:26:28.714743+0000","last_clean":"2023-10-21T05:26:28.714743+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:26:28.714743+0000","last_undegraded":"2023-10-21T05:26:28.714743+0000","last_fullsized":"2023-10-21T05:26:28.714743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30884126739,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2423263,"num_read_kb":9578120,"num_write":2421315,"num_write_kb":9685254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19836993536,"data_stored":39440810022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40253,"ondisk_log_size":40253,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":48915,"ondisk_log_size":48915,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739432,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11519804,"kb_used_data":9687412,"kb_used_omap":1519,"kb_used_meta":1830864,"kb_avail":82852036,"statfs":{"total":96636764160,"available":84840484864,"internally_reserved":0,"allocated":9919909888,"data_stored":19720778810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1556259,"internal_metadata":1874804957},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:31 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.83299999999999996,"15min":0.997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":4.0309999999999997,"5min":5.5609999999999999,"15min":27.184999999999999},"last":0.317},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.86099999999999999,"15min":0.997},"min":{"1min":0.26600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":2.468,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:27:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.84899999999999998,"15min":0.88400000000000001},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":4.0810000000000004,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":1,"5min":0.83199999999999996,"15min":0.95599999999999996},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":4.1159999999999997,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739430,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6178556,"kb_used_data":4757428,"kb_used_omap":1339,"kb_used_meta":1419780,"kb_avail":88193284,"statfs":{"total":96636764160,"available":90309922816,"internally_reserved":0,"allocated":4871606272,"data_stored":9682641934,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1453855068},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,5,10,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:24 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.72499999999999998,"15min":0.75700000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":3.665,"5min":4.5010000000000003,"15min":5.3920000000000003},"last":0.879},{"interface":"front","average":{"1min":1.4379999999999999,"5min":0.90100000000000002,"15min":0.82299999999999995},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.968,"5min":6.968,"15min":6.968},"last":1.7290000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:27:24 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.014,"15min":0.91200000000000003},"min":{"1min":0.26000000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":3.339,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.024,"15min":0.92100000000000004},"min":{"1min":0.25,"5min":0.222,"15min":0.22},"max":{"1min":2.5419999999999998,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5834592,"kb_used_data":4930836,"kb_used_omap":1499,"kb_used_meta":902244,"kb_avail":88537248,"statfs":{"total":96636764160,"available":90662141952,"internally_reserved":0,"allocated":5049176064,"data_stored":10038557925,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":923898047},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,11,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:27:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.752,"15min":0.76800000000000002},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.3089999999999999,"5min":5.923,"15min":5.923},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.77200000000000002,"15min":0.80900000000000005},"min":{"1min":0.28799999999999998,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":1.5669999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.36799999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:27:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1240000000000001,"15min":0.97099999999999997},"min":{"1min":0.254,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":4.6120000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"last":4.8529999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.98099999999999998,"15min":0.95799999999999996},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.95,"5min":3.9089999999999998,"15min":6.1779999999999999},"last":0.92200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5047402496,"data_stored":10037448704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4870299648,"data_stored":9682161683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9919291392,"data_stored":19721199635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:28:08.989 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:28:08.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:28:09.523 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:28:09.542 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:28:28.625 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:28:28.620+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:28:28.625 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:28:28.620+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:28:29.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:28:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:28:30.150 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:28:30.171 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2816,"stamp":"2023-10-21T05:28:30.007985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30928647933,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2483256,"num_read_kb":9661258,"num_write":2433204,"num_write_kb":9730591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89425,"ondisk_log_size":89425,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24065584,"kb_used_data":19461656,"kb_used_omap":4359,"kb_used_meta":4599544,"kb_avail":259049936,"statfs":{"total":289910292480,"available":265267134464,"internally_reserved":0,"allocated":19928735744,"data_stored":39530022189,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4463880,"internal_metadata":4709933816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,10,4,9,11,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26677248,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6355,"num_read_kb":25376,"num_write":6517,"num_write_kb":26068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001967"},"pg_stats":[{"pgid":"2.7","version":"153'8092","reported_seq":11793,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:28.552916+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:28:28.552916+0000","last_peered":"2023-10-21T05:28:28.552916+0000","last_clean":"2023-10-21T05:28:28.552916+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:28:28.552916+0000","last_undegraded":"2023-10-21T05:28:28.552916+0000","last_fullsized":"2023-10-21T05:28:28.552916+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'4140","reported_seq":8037,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:27.847092+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:28:27.847092+0000","last_peered":"2023-10-21T05:28:27.847092+0000","last_clean":"2023-10-21T05:28:27.847092+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:28:27.847092+0000","last_undegraded":"2023-10-21T05:28:27.847092+0000","last_fullsized":"2023-10-21T05:28:27.847092+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'12080","reported_seq":17487,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:28.564151+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:28:28.564151+0000","last_peered":"2023-10-21T05:28:28.564151+0000","last_clean":"2023-10-21T05:28:28.564151+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:28:28.564151+0000","last_undegraded":"2023-10-21T05:28:28.564151+0000","last_fullsized":"2023-10-21T05:28:28.564151+0000","mapping_epoch":11,"log_start":"66'2000","ondisk_log_start":"66'2000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'8137","reported_seq":12853,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:28.547643+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:28:28.547643+0000","last_peered":"2023-10-21T05:28:28.547643+0000","last_clean":"2023-10-21T05:28:28.547643+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:28:28.547643+0000","last_undegraded":"2023-10-21T05:28:28.547643+0000","last_fullsized":"2023-10-21T05:28:28.547643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'611846","reported_seq":1209773,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:28.252538+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:28:28.252538+0000","last_peered":"2023-10-21T05:28:28.252538+0000","last_clean":"2023-10-21T05:28:28.252538+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:28:28.252538+0000","last_undegraded":"2023-10-21T05:28:28.252538+0000","last_fullsized":"2023-10-21T05:28:28.252538+0000","mapping_epoch":14,"log_start":"152'601762","ondisk_log_start":"152'601762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7648550912,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":602224,"num_read_kb":2380332,"num_write":601691,"num_write_kb":2406764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'16432","reported_seq":23778,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:27.777094+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:28:27.777094+0000","last_peered":"2023-10-21T05:28:27.777094+0000","last_clean":"2023-10-21T05:28:27.777094+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:28:27.777094+0000","last_undegraded":"2023-10-21T05:28:27.777094+0000","last_fullsized":"2023-10-21T05:28:27.777094+0000","mapping_epoch":11,"log_start":"101'6400","ondisk_log_start":"101'6400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'619030","reported_seq":1225387,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:29.452624+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:28:29.452624+0000","last_peered":"2023-10-21T05:28:29.452624+0000","last_clean":"2023-10-21T05:28:29.452624+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:28:29.452624+0000","last_undegraded":"2023-10-21T05:28:29.452624+0000","last_fullsized":"2023-10-21T05:28:29.452624+0000","mapping_epoch":14,"log_start":"152'608961","ondisk_log_start":"152'608961","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7743213568,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":610690,"num_read_kb":2413348,"num_write":608789,"num_write_kb":2435156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3838,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:27.787283+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:28:27.787283+0000","last_peered":"2023-10-21T05:28:27.787283+0000","last_clean":"2023-10-21T05:28:27.787283+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:28:27.787283+0000","last_undegraded":"2023-10-21T05:28:27.787283+0000","last_fullsized":"2023-10-21T05:28:27.787283+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'646830","reported_seq":1276643,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:28.242733+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:28:28.242733+0000","last_peered":"2023-10-21T05:28:28.242733+0000","last_clean":"2023-10-21T05:28:28.242733+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:28:28.242733+0000","last_undegraded":"2023-10-21T05:28:28.242733+0000","last_fullsized":"2023-10-21T05:28:28.242733+0000","mapping_epoch":14,"log_start":"152'636754","ondisk_log_start":"152'636754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8088010752,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":634330,"num_read_kb":2507592,"num_write":636116,"num_write_kb":2544464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7624,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:27.779427+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:28:27.779427+0000","last_peered":"2023-10-21T05:28:27.779427+0000","last_clean":"2023-10-21T05:28:27.779427+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:28:27.779427+0000","last_undegraded":"2023-10-21T05:28:27.779427+0000","last_fullsized":"2023-10-21T05:28:27.779427+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'593698","reported_seq":1176202,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:29.454772+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:28:29.454772+0000","last_peered":"2023-10-21T05:28:29.454772+0000","last_clean":"2023-10-21T05:28:29.454772+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:28:29.454772+0000","last_undegraded":"2023-10-21T05:28:29.454772+0000","last_fullsized":"2023-10-21T05:28:29.454772+0000","mapping_epoch":14,"log_start":"152'583660","ondisk_log_start":"152'583660","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4142,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 638s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7448277011,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":586553,"num_read_kb":2318916,"num_write":585450,"num_write_kb":2341794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'8049","reported_seq":11682,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:28.557860+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:28:28.557860+0000","last_peered":"2023-10-21T05:28:28.557860+0000","last_clean":"2023-10-21T05:28:28.557860+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:28:28.557860+0000","last_undegraded":"2023-10-21T05:28:28.557860+0000","last_fullsized":"2023-10-21T05:28:28.557860+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8049,"log_dups_size":0,"ondisk_log_size":8049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":335,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:39.609073+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:27:39.609073+0000","last_peered":"2023-10-21T05:27:39.609073+0000","last_clean":"2023-10-21T05:27:39.609073+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:27:39.609073+0000","last_undegraded":"2023-10-21T05:27:39.609073+0000","last_fullsized":"2023-10-21T05:27:39.609073+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30928052243,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2433797,"num_read_kb":9620188,"num_write":2432046,"num_write_kb":9728178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19924795392,"data_stored":39528611878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40267,"ondisk_log_size":40267,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49059,"ondisk_log_size":49059,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739437,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11784624,"kb_used_data":9730088,"kb_used_omap":1519,"kb_used_meta":2053008,"kb_avail":82587216,"statfs":{"total":96636764160,"available":84569309184,"internally_reserved":0,"allocated":9963610112,"data_stored":19764479034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1556259,"internal_metadata":2102280413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:31 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.83299999999999996,"15min":0.997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":4.0309999999999997,"5min":5.5609999999999999,"15min":27.184999999999999},"last":1.9379999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.86099999999999999,"15min":0.997},"min":{"1min":0.26600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":2.468,"5min":5.5229999999999997,"15min":27.251000000000001},"last":1.9850000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:27:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.84899999999999998,"15min":0.88400000000000001},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":4.0810000000000004,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.249},{"interface":"front","average":{"1min":1,"5min":0.83199999999999996,"15min":0.95599999999999996},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":4.1159999999999997,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":2.7519999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739435,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6326028,"kb_used_data":4778948,"kb_used_omap":1339,"kb_used_meta":1545732,"kb_avail":88045812,"statfs":{"total":96636764160,"available":90158911488,"internally_reserved":0,"allocated":4893642752,"data_stored":9704678414,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1582829916},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,2,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.434},{"interface":"front","average":{"1min":1.075,"5min":0.94599999999999995,"15min":0.85099999999999998},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.5019999999999998,"5min":6.968,"15min":6.968},"last":0.66700000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.98799999999999999,"15min":0.89900000000000002},"min":{"1min":0.22,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.5410000000000004,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.016,"15min":0.91700000000000004},"min":{"1min":0.28199999999999997,"5min":0.222,"15min":0.22},"max":{"1min":4.7270000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5954932,"kb_used_data":4952620,"kb_used_omap":1499,"kb_used_meta":1000804,"kb_avail":88416908,"statfs":{"total":96636764160,"available":90538913792,"internally_reserved":0,"allocated":5071482880,"data_stored":10060864741,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1024823487},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:27:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.752,"15min":0.76800000000000002},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.3089999999999999,"5min":5.923,"15min":5.923},"last":0.379},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.77200000000000002,"15min":0.80900000000000005},"min":{"1min":0.28799999999999998,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":1.5669999999999999,"5min":5.8879999999999999,"15min":6.4050000000000002},"last":0.41599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:27:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1240000000000001,"15min":0.97099999999999997},"min":{"1min":0.254,"5min":0.214,"15min":0.20100000000000001},"max":{"1min":4.6120000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"last":1.532},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.98099999999999998,"15min":0.95799999999999996},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.95,"5min":3.9089999999999998,"15min":6.1779999999999999},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5070061568,"data_stored":10060107776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4891598848,"data_stored":9703460883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9963134976,"data_stored":19765043219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:28:30.173 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:28:30.173 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:28:30.656 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.652+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:28:30.672 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.666+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:28:30.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.681+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:28:30.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.681+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:28:30.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.681+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:28:30.688 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.681+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:28:30.689 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:30.681+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:28:32.994 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:32.990+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:28:32.994 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:32.990+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:28:32.994 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:28:32.990+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:28:35.004 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:28:35.001+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:28:50.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:28:50.373 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:28:50.369+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:28:50.698 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:28:50.698 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:28:50.720 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2827,"stamp":"2023-10-21T05:28:50.011320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30963615485,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2491946,"num_read_kb":9695958,"num_write":2441747,"num_write_kb":9764763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89346,"ondisk_log_size":89346,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23980344,"kb_used_data":19529496,"kb_used_omap":4360,"kb_used_meta":4446455,"kb_avail":259135176,"statfs":{"total":289910292480,"available":265354420224,"internally_reserved":0,"allocated":19998203904,"data_stored":39599490349,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4465560,"internal_metadata":4553170024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,11,3,25,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17969152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4439,"num_read_kb":17736,"num_write":4393,"num_write_kb":17572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001722"},"pg_stats":[{"pgid":"2.7","version":"153'8119","reported_seq":11828,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.258646+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:28:48.258646+0000","last_peered":"2023-10-21T05:28:48.258646+0000","last_clean":"2023-10-21T05:28:48.258646+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:28:48.258646+0000","last_undegraded":"2023-10-21T05:28:48.258646+0000","last_fullsized":"2023-10-21T05:28:48.258646+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'4154","reported_seq":8055,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:46.466479+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:28:46.466479+0000","last_peered":"2023-10-21T05:28:46.466479+0000","last_clean":"2023-10-21T05:28:46.466479+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:28:46.466479+0000","last_undegraded":"2023-10-21T05:28:46.466479+0000","last_fullsized":"2023-10-21T05:28:46.466479+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4154,"log_dups_size":0,"ondisk_log_size":4154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'12120","reported_seq":17539,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.258207+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:28:48.258207+0000","last_peered":"2023-10-21T05:28:48.258207+0000","last_clean":"2023-10-21T05:28:48.258207+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:28:48.258207+0000","last_undegraded":"2023-10-21T05:28:48.258207+0000","last_fullsized":"2023-10-21T05:28:48.258207+0000","mapping_epoch":11,"log_start":"66'2100","ondisk_log_start":"66'2100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'8163","reported_seq":12887,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.258288+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:28:48.258288+0000","last_peered":"2023-10-21T05:28:48.258288+0000","last_clean":"2023-10-21T05:28:48.258288+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:28:48.258288+0000","last_undegraded":"2023-10-21T05:28:48.258288+0000","last_fullsized":"2023-10-21T05:28:48.258288+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'614106","reported_seq":1214183,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.416426+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:28:48.416426+0000","last_peered":"2023-10-21T05:28:48.416426+0000","last_clean":"2023-10-21T05:28:48.416426+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:28:48.416426+0000","last_undegraded":"2023-10-21T05:28:48.416426+0000","last_fullsized":"2023-10-21T05:28:48.416426+0000","mapping_epoch":14,"log_start":"152'604062","ondisk_log_start":"152'604062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7657803776,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":604372,"num_read_kb":2388912,"num_write":603951,"num_write_kb":2415804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'16467","reported_seq":23829,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.257027+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:28:48.257027+0000","last_peered":"2023-10-21T05:28:48.257027+0000","last_clean":"2023-10-21T05:28:48.257027+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:28:48.257027+0000","last_undegraded":"2023-10-21T05:28:48.257027+0000","last_fullsized":"2023-10-21T05:28:48.257027+0000","mapping_epoch":11,"log_start":"101'6400","ondisk_log_start":"101'6400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'621168","reported_seq":1229661,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:49.608383+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:28:49.608383+0000","last_peered":"2023-10-21T05:28:49.608383+0000","last_clean":"2023-10-21T05:28:49.608383+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:28:49.608383+0000","last_undegraded":"2023-10-21T05:28:49.608383+0000","last_fullsized":"2023-10-21T05:28:49.608383+0000","mapping_epoch":14,"log_start":"152'611161","ondisk_log_start":"152'611161","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7751950336,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":612828,"num_read_kb":2421880,"num_write":610927,"num_write_kb":2443708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3847,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.259933+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:28:48.259933+0000","last_peered":"2023-10-21T05:28:48.259933+0000","last_clean":"2023-10-21T05:28:48.259933+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:28:48.259933+0000","last_undegraded":"2023-10-21T05:28:48.259933+0000","last_fullsized":"2023-10-21T05:28:48.259933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'649064","reported_seq":1281211,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.392495+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:28:48.392495+0000","last_peered":"2023-10-21T05:28:48.392495+0000","last_clean":"2023-10-21T05:28:48.392495+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:28:48.392495+0000","last_undegraded":"2023-10-21T05:28:48.392495+0000","last_fullsized":"2023-10-21T05:28:48.392495+0000","mapping_epoch":14,"log_start":"152'639054","ondisk_log_start":"152'639054","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8097173504,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":636662,"num_read_kb":2516908,"num_write":638350,"num_write_kb":2553400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7641,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.259829+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:28:48.259829+0000","last_peered":"2023-10-21T05:28:48.259829+0000","last_clean":"2023-10-21T05:28:48.259829+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:28:48.259829+0000","last_undegraded":"2023-10-21T05:28:48.259829+0000","last_fullsized":"2023-10-21T05:28:48.259829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'595719","reported_seq":1180309,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:49.607317+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:28:49.607317+0000","last_peered":"2023-10-21T05:28:49.607317+0000","last_clean":"2023-10-21T05:28:49.607317+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:28:49.607317+0000","last_undegraded":"2023-10-21T05:28:49.607317+0000","last_fullsized":"2023-10-21T05:28:49.607317+0000","mapping_epoch":14,"log_start":"152'585660","ondisk_log_start":"152'585660","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4279,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 658s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7456092179,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":588625,"num_read_kb":2327188,"num_write":587361,"num_write_kb":2349438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'8075","reported_seq":11716,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:48.258400+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:28:48.258400+0000","last_peered":"2023-10-21T05:28:48.258400+0000","last_clean":"2023-10-21T05:28:48.258400+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:28:48.258400+0000","last_undegraded":"2023-10-21T05:28:48.258400+0000","last_fullsized":"2023-10-21T05:28:48.258400+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":335,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:27:39.609073+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:27:39.609073+0000","last_peered":"2023-10-21T05:27:39.609073+0000","last_clean":"2023-10-21T05:27:39.609073+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:27:39.609073+0000","last_undegraded":"2023-10-21T05:27:39.609073+0000","last_fullsized":"2023-10-21T05:27:39.609073+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30963019795,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2442487,"num_read_kb":9654888,"num_write":2440589,"num_write_kb":9762350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19994619904,"data_stored":39598436390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40120,"ondisk_log_size":40120,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49127,"ondisk_log_size":49127,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739441,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11476864,"kb_used_data":9763764,"kb_used_omap":1521,"kb_used_meta":1711566,"kb_avail":82894976,"statfs":{"total":96636764160,"available":84884455424,"internally_reserved":0,"allocated":9998094336,"data_stored":19798963258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1557939,"internal_metadata":1752644173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":0.93899999999999995,"15min":1.0009999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.186},"max":{"1min":8.3439999999999994,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.624},{"interface":"front","average":{"1min":1.216,"5min":0.97299999999999998,"15min":1.022},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.191},"max":{"1min":3.3900000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.27200000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:28:35 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":0.97299999999999998,"15min":0.89500000000000002},"min":{"1min":0.249,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":8.2710000000000008,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.5800000000000001,"5min":0.98999999999999999,"15min":0.98399999999999999},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.20399999999999999},"max":{"1min":8.3100000000000005,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.66200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739439,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6422272,"kb_used_data":4795124,"kb_used_omap":1339,"kb_used_meta":1625796,"kb_avail":87949568,"statfs":{"total":96636764160,"available":90060357632,"internally_reserved":0,"allocated":4910206976,"data_stored":9721242638,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1664815452},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,14,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.075,"5min":0.94599999999999995,"15min":0.85099999999999998},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.5019999999999998,"5min":6.968,"15min":6.968},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.98799999999999999,"15min":0.89900000000000002},"min":{"1min":0.22,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.5410000000000004,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":1.296},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.016,"15min":0.91700000000000004},"min":{"1min":0.28199999999999997,"5min":0.222,"15min":0.22},"max":{"1min":4.7270000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":1.621}]}]},{"osd":0,"up_from":8,"seq":34359739439,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6081208,"kb_used_data":4970608,"kb_used_omap":1499,"kb_used_meta":1109092,"kb_avail":88290632,"statfs":{"total":96636764160,"available":90409607168,"internally_reserved":0,"allocated":5089902592,"data_stored":10079284453,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1135710399},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.74199999999999999,"15min":0.78100000000000003},"min":{"1min":0.28399999999999997,"5min":0.187,"15min":0.187},"max":{"1min":2.9220000000000002,"5min":4.8090000000000002,"15min":5.923},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":0.76800000000000002,"15min":0.81699999999999995},"min":{"1min":0.32100000000000001,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":2.9550000000000001,"5min":4.8639999999999999,"15min":6.4050000000000002},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:28:39 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.8529999999999998,"5min":4.867,"15min":14.053000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.046,"5min":0.94799999999999995,"15min":0.97099999999999997},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.8439999999999999,"5min":3.9089999999999998,"15min":6.1779999999999999},"last":1.161}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5088481280,"data_stored":10078527488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4908191744,"data_stored":9720053779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9997946880,"data_stored":19799855123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:28:50.722 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:28:50.722 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:10.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:29:11.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:29:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:29:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2837,"stamp":"2023-10-21T05:29:10.016309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30998337277,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2500226,"num_read_kb":9729026,"num_write":2450218,"num_write_kb":9798647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89498,"ondisk_log_size":89498,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24219332,"kb_used_data":19596780,"kb_used_omap":4360,"kb_used_meta":4618167,"kb_avail":258896188,"statfs":{"total":289910292480,"available":265109696512,"internally_reserved":0,"allocated":20067102720,"data_stored":39668376994,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4465560,"internal_metadata":4729003112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,4,4,15,15,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25980928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6120,"num_read_kb":24436,"num_write":6341,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003488"},"pg_stats":[{"pgid":"2.7","version":"154'8145","reported_seq":11863,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:09.114878+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:29:09.114878+0000","last_peered":"2023-10-21T05:29:09.114878+0000","last_clean":"2023-10-21T05:29:09.114878+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:29:09.114878+0000","last_undegraded":"2023-10-21T05:29:09.114878+0000","last_fullsized":"2023-10-21T05:29:09.114878+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'4167","reported_seq":8073,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:08.736741+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:29:08.736741+0000","last_peered":"2023-10-21T05:29:08.736741+0000","last_clean":"2023-10-21T05:29:08.736741+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:29:08.736741+0000","last_undegraded":"2023-10-21T05:29:08.736741+0000","last_fullsized":"2023-10-21T05:29:08.736741+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'12159","reported_seq":17591,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:09.116481+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:29:09.116481+0000","last_peered":"2023-10-21T05:29:09.116481+0000","last_clean":"2023-10-21T05:29:09.116481+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:29:09.116481+0000","last_undegraded":"2023-10-21T05:29:09.116481+0000","last_fullsized":"2023-10-21T05:29:09.116481+0000","mapping_epoch":11,"log_start":"66'2100","ondisk_log_start":"66'2100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'8189","reported_seq":12922,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:09.117231+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:29:09.117231+0000","last_peered":"2023-10-21T05:29:09.117231+0000","last_clean":"2023-10-21T05:29:09.117231+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:29:09.117231+0000","last_undegraded":"2023-10-21T05:29:09.117231+0000","last_fullsized":"2023-10-21T05:29:09.117231+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'616216","reported_seq":1218297,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:08.579370+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:29:08.579370+0000","last_peered":"2023-10-21T05:29:08.579370+0000","last_clean":"2023-10-21T05:29:08.579370+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:29:08.579370+0000","last_undegraded":"2023-10-21T05:29:08.579370+0000","last_fullsized":"2023-10-21T05:29:08.579370+0000","mapping_epoch":14,"log_start":"152'606162","ondisk_log_start":"152'606162","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7666442240,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":606374,"num_read_kb":2396904,"num_write":606061,"num_write_kb":2424244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"154'16535","reported_seq":23910,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:05.961024+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:29:05.961024+0000","last_peered":"2023-10-21T05:29:05.961024+0000","last_clean":"2023-10-21T05:29:05.961024+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:29:05.961024+0000","last_undegraded":"2023-10-21T05:29:05.961024+0000","last_fullsized":"2023-10-21T05:29:05.961024+0000","mapping_epoch":11,"log_start":"101'6500","ondisk_log_start":"101'6500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'623293","reported_seq":1233876,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:09.830690+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:29:09.830690+0000","last_peered":"2023-10-21T05:29:09.830690+0000","last_clean":"2023-10-21T05:29:09.830690+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:29:09.830690+0000","last_undegraded":"2023-10-21T05:29:09.830690+0000","last_fullsized":"2023-10-21T05:29:09.830690+0000","mapping_epoch":14,"log_start":"152'613261","ondisk_log_start":"152'613261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7760666624,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":614920,"num_read_kb":2430228,"num_write":613052,"num_write_kb":2452208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3855,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:08.740376+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:29:08.740376+0000","last_peered":"2023-10-21T05:29:08.740376+0000","last_clean":"2023-10-21T05:29:08.740376+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:29:08.740376+0000","last_undegraded":"2023-10-21T05:29:08.740376+0000","last_fullsized":"2023-10-21T05:29:08.740376+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'651266","reported_seq":1285575,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:08.565943+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:29:08.565943+0000","last_peered":"2023-10-21T05:29:08.565943+0000","last_clean":"2023-10-21T05:29:08.565943+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:29:08.565943+0000","last_undegraded":"2023-10-21T05:29:08.565943+0000","last_fullsized":"2023-10-21T05:29:08.565943+0000","mapping_epoch":14,"log_start":"152'641254","ondisk_log_start":"152'641254","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8106209280,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":638823,"num_read_kb":2525544,"num_write":640552,"num_write_kb":2562208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7657,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:08.738291+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:29:08.738291+0000","last_peered":"2023-10-21T05:29:08.738291+0000","last_clean":"2023-10-21T05:29:08.738291+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:29:08.738291+0000","last_undegraded":"2023-10-21T05:29:08.738291+0000","last_fullsized":"2023-10-21T05:29:08.738291+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'597836","reported_seq":1184458,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:09.824700+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:29:09.824700+0000","last_peered":"2023-10-21T05:29:09.824700+0000","last_clean":"2023-10-21T05:29:09.824700+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:29:09.824700+0000","last_undegraded":"2023-10-21T05:29:09.824700+0000","last_fullsized":"2023-10-21T05:29:09.824700+0000","mapping_epoch":14,"log_start":"152'587760","ondisk_log_start":"152'587760","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4380,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 678s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7464423443,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":590650,"num_read_kb":2335280,"num_write":589395,"num_write_kb":2357574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'8101","reported_seq":11751,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:09.119784+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:29:09.119784+0000","last_peered":"2023-10-21T05:29:09.119784+0000","last_clean":"2023-10-21T05:29:09.119784+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:29:09.119784+0000","last_undegraded":"2023-10-21T05:29:09.119784+0000","last_fullsized":"2023-10-21T05:29:09.119784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8101,"log_dups_size":0,"ondisk_log_size":8101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":336,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:29.462110+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:28:29.462110+0000","last_peered":"2023-10-21T05:28:29.462110+0000","last_clean":"2023-10-21T05:28:29.462110+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:28:29.462110+0000","last_undegraded":"2023-10-21T05:28:29.462110+0000","last_fullsized":"2023-10-21T05:28:29.462110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30997741587,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2450767,"num_read_kb":9687956,"num_write":2449060,"num_write_kb":9796234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20063817728,"data_stored":39667634214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40174,"ondisk_log_size":40174,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49225,"ondisk_log_size":49225,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739445,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11511708,"kb_used_data":9797396,"kb_used_omap":1521,"kb_used_meta":1712782,"kb_avail":82860132,"statfs":{"total":96636764160,"available":84848775168,"internally_reserved":0,"allocated":10032533504,"data_stored":19833398369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1557939,"internal_metadata":1753889357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":0.93899999999999995,"15min":1.0009999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.186},"max":{"1min":8.3439999999999994,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.216,"5min":0.97299999999999998,"15min":1.022},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.191},"max":{"1min":3.3900000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:28:35 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":0.97299999999999998,"15min":0.89500000000000002},"min":{"1min":0.249,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":8.2710000000000008,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.5800000000000001,"5min":0.98999999999999999,"15min":0.98399999999999999},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.20399999999999999},"max":{"1min":8.3100000000000005,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359739443,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6511116,"kb_used_data":4811780,"kb_used_omap":1339,"kb_used_meta":1697988,"kb_avail":87860724,"statfs":{"total":96636764160,"available":89969381376,"internally_reserved":0,"allocated":4927262720,"data_stored":9738294325,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1738740060},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,11,5,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.075,"5min":0.94599999999999995,"15min":0.85099999999999998},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.5019999999999998,"5min":6.968,"15min":6.968},"last":0.254}]},{"osd":2,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.98799999999999999,"15min":0.89900000000000002},"min":{"1min":0.22,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.5410000000000004,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.016,"15min":0.91700000000000004},"min":{"1min":0.28199999999999997,"5min":0.222,"15min":0.22},"max":{"1min":4.7270000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.47399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6196508,"kb_used_data":4987604,"kb_used_omap":1499,"kb_used_meta":1207396,"kb_avail":88175332,"statfs":{"total":96636764160,"available":90291539968,"internally_reserved":0,"allocated":5107306496,"data_stored":10096684300,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1236373695},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.74199999999999999,"15min":0.78100000000000003},"min":{"1min":0.28399999999999997,"5min":0.187,"15min":0.187},"max":{"1min":2.9220000000000002,"5min":4.8090000000000002,"15min":5.923},"last":0.314},{"interface":"front","average":{"1min":1.0509999999999999,"5min":0.76800000000000002,"15min":0.81699999999999995},"min":{"1min":0.32100000000000001,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":2.9550000000000001,"5min":4.8639999999999999,"15min":6.4050000000000002},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:28:39 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.8529999999999998,"5min":4.867,"15min":14.053000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.046,"5min":0.94799999999999995,"15min":0.97099999999999997},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.8439999999999999,"5min":3.9089999999999998,"15min":6.1779999999999999},"last":2.238}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5106126848,"data_stored":10096173056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4925218816,"data_stored":9737080851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10032472064,"data_stored":19834380307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:29:11.272 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:29:11.272 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:19.313 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:29:19.310+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:29:29.877 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:29:29.873+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:29:29.878 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:29:29.873+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:29:31.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:29:31.838 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:29:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:29:31.858 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2847,"stamp":"2023-10-21T05:29:30.019673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31028500221,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2507657,"num_read_kb":9758682,"num_write":2457588,"num_write_kb":9828127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89544,"ondisk_log_size":89544,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24494900,"kb_used_data":19651612,"kb_used_omap":4360,"kb_used_meta":4838903,"kb_avail":258620620,"statfs":{"total":289910292480,"available":264827514880,"internally_reserved":0,"allocated":20123250688,"data_stored":39724524962,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4465560,"internal_metadata":4955036776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,7,8,4,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21729280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5261,"num_read_kb":20996,"num_write":5310,"num_write_kb":21240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002001"},"pg_stats":[{"pgid":"2.7","version":"154'8173","reported_seq":11898,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.352615+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:29:24.352615+0000","last_peered":"2023-10-21T05:29:24.352615+0000","last_clean":"2023-10-21T05:29:24.352615+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:29:24.352615+0000","last_undegraded":"2023-10-21T05:29:24.352615+0000","last_fullsized":"2023-10-21T05:29:24.352615+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8173,"log_dups_size":0,"ondisk_log_size":8173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'4181","reported_seq":8090,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.098352+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:29:24.098352+0000","last_peered":"2023-10-21T05:29:24.098352+0000","last_clean":"2023-10-21T05:29:24.098352+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:29:24.098352+0000","last_undegraded":"2023-10-21T05:29:24.098352+0000","last_fullsized":"2023-10-21T05:29:24.098352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'12201","reported_seq":17642,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.366513+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:29:24.366513+0000","last_peered":"2023-10-21T05:29:24.366513+0000","last_clean":"2023-10-21T05:29:24.366513+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:29:24.366513+0000","last_undegraded":"2023-10-21T05:29:24.366513+0000","last_fullsized":"2023-10-21T05:29:24.366513+0000","mapping_epoch":11,"log_start":"68'2200","ondisk_log_start":"68'2200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'8217","reported_seq":12956,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.352426+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:29:24.352426+0000","last_peered":"2023-10-21T05:29:24.352426+0000","last_clean":"2023-10-21T05:29:24.352426+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:29:24.352426+0000","last_undegraded":"2023-10-21T05:29:24.352426+0000","last_fullsized":"2023-10-21T05:29:24.352426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'618272","reported_seq":1222438,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:28.677143+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:29:28.677143+0000","last_peered":"2023-10-21T05:29:28.677143+0000","last_clean":"2023-10-21T05:29:28.677143+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:29:28.677143+0000","last_undegraded":"2023-10-21T05:29:28.677143+0000","last_fullsized":"2023-10-21T05:29:28.677143+0000","mapping_epoch":14,"log_start":"153'608262","ondisk_log_start":"153'608262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7674880000,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":608462,"num_read_kb":2405232,"num_write":608117,"num_write_kb":2432468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"154'16591","reported_seq":23982,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:28.581106+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:29:28.581106+0000","last_peered":"2023-10-21T05:29:28.581106+0000","last_clean":"2023-10-21T05:29:28.581106+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:29:28.581106+0000","last_undegraded":"2023-10-21T05:29:28.581106+0000","last_fullsized":"2023-10-21T05:29:28.581106+0000","mapping_epoch":11,"log_start":"101'6500","ondisk_log_start":"101'6500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'624898","reported_seq":1237045,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.927035+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:29:24.927035+0000","last_peered":"2023-10-21T05:29:24.927035+0000","last_clean":"2023-10-21T05:29:24.927035+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:29:24.927035+0000","last_undegraded":"2023-10-21T05:29:24.927035+0000","last_fullsized":"2023-10-21T05:29:24.927035+0000","mapping_epoch":14,"log_start":"153'614861","ondisk_log_start":"153'614861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7767236608,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":616477,"num_read_kb":2436444,"num_write":614657,"num_write_kb":2458628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3861,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.098346+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:29:24.098346+0000","last_peered":"2023-10-21T05:29:24.098346+0000","last_clean":"2023-10-21T05:29:24.098346+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:29:24.098346+0000","last_undegraded":"2023-10-21T05:29:24.098346+0000","last_fullsized":"2023-10-21T05:29:24.098346+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'653481","reported_seq":1290057,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:28.652419+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:29:28.652419+0000","last_peered":"2023-10-21T05:29:28.652419+0000","last_clean":"2023-10-21T05:29:28.652419+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:29:28.652419+0000","last_undegraded":"2023-10-21T05:29:28.652419+0000","last_fullsized":"2023-10-21T05:29:28.652419+0000","mapping_epoch":14,"log_start":"153'643454","ondisk_log_start":"153'643454","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8115261440,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":641089,"num_read_kb":2534584,"num_write":642767,"num_write_kb":2571068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7669,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.098287+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:29:24.098287+0000","last_peered":"2023-10-21T05:29:24.098287+0000","last_clean":"2023-10-21T05:29:24.098287+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:29:24.098287+0000","last_undegraded":"2023-10-21T05:29:24.098287+0000","last_fullsized":"2023-10-21T05:29:24.098287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'599410","reported_seq":1187563,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:24.939272+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:29:24.939272+0000","last_peered":"2023-10-21T05:29:24.939272+0000","last_clean":"2023-10-21T05:29:24.939272+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:29:24.939272+0000","last_undegraded":"2023-10-21T05:29:24.939272+0000","last_fullsized":"2023-10-21T05:29:24.939272+0000","mapping_epoch":14,"log_start":"153'589360","ondisk_log_start":"153'589360","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4477,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 693s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7470526483,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":592170,"num_read_kb":2341352,"num_write":590889,"num_write_kb":2363550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'8129","reported_seq":11785,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:24.366279+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:29:24.366279+0000","last_peered":"2023-10-21T05:29:24.366279+0000","last_clean":"2023-10-21T05:29:24.366279+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:29:24.366279+0000","last_undegraded":"2023-10-21T05:29:24.366279+0000","last_fullsized":"2023-10-21T05:29:24.366279+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8129,"log_dups_size":0,"ondisk_log_size":8129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":336,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:29.462110+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:28:29.462110+0000","last_peered":"2023-10-21T05:28:29.462110+0000","last_clean":"2023-10-21T05:28:29.462110+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:28:29.462110+0000","last_undegraded":"2023-10-21T05:28:29.462110+0000","last_fullsized":"2023-10-21T05:28:29.462110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31027904531,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2458198,"num_read_kb":9717612,"num_write":2456430,"num_write_kb":9825714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20120121344,"data_stored":39723937830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40124,"ondisk_log_size":40124,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49321,"ondisk_log_size":49321,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739448,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11619028,"kb_used_data":9822796,"kb_used_omap":1521,"kb_used_meta":1794702,"kb_avail":82752812,"statfs":{"total":96636764160,"available":84738879488,"internally_reserved":0,"allocated":10058543104,"data_stored":19859407969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1557939,"internal_metadata":1837775437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":0.93899999999999995,"15min":1.0009999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.186},"max":{"1min":8.3439999999999994,"5min":8.3439999999999994,"15min":27.184999999999999},"last":1.913},{"interface":"front","average":{"1min":1.216,"5min":0.97299999999999998,"15min":1.022},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.191},"max":{"1min":3.3900000000000001,"5min":5.5229999999999997,"15min":27.251000000000001},"last":0.31}]},{"osd":1,"last update":"Sat Oct 21 05:28:35 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":0.97299999999999998,"15min":0.89500000000000002},"min":{"1min":0.249,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":8.2710000000000008,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":0.98999999999999999,"15min":0.98399999999999999},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.20399999999999999},"max":{"1min":8.3100000000000005,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":1.8759999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739446,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6572004,"kb_used_data":4824092,"kb_used_omap":1339,"kb_used_meta":1746564,"kb_avail":87799836,"statfs":{"total":96636764160,"available":89907032064,"internally_reserved":0,"allocated":4939870208,"data_stored":9750901813,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1788481884},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,5,6,3,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.245},{"interface":"front","average":{"1min":1.075,"5min":0.94599999999999995,"15min":0.85099999999999998},"min":{"1min":0.23300000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.5019999999999998,"5min":6.968,"15min":6.968},"last":0.20999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.98799999999999999,"15min":0.89900000000000002},"min":{"1min":0.22,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":5.5410000000000004,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.016,"15min":0.91700000000000004},"min":{"1min":0.28199999999999997,"5min":0.222,"15min":0.22},"max":{"1min":4.7270000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6303868,"kb_used_data":5004724,"kb_used_omap":1499,"kb_used_meta":1297636,"kb_avail":88067972,"statfs":{"total":96636764160,"available":90181603328,"internally_reserved":0,"allocated":5124837376,"data_stored":10114215180,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1328779455},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.74199999999999999,"15min":0.78100000000000003},"min":{"1min":0.28399999999999997,"5min":0.187,"15min":0.187},"max":{"1min":2.9220000000000002,"5min":4.8090000000000002,"15min":5.923},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":0.76800000000000002,"15min":0.81699999999999995},"min":{"1min":0.32100000000000001,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":2.9550000000000001,"5min":4.8639999999999999,"15min":6.4050000000000002},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:28:39 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.8529999999999998,"5min":4.867,"15min":14.053000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.046,"5min":0.94799999999999995,"15min":0.97099999999999997},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.8439999999999999,"5min":3.9089999999999998,"15min":6.1779999999999999},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5123633152,"data_stored":10113679360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4937912320,"data_stored":9749774355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10058575872,"data_stored":19860484115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:29:31.860 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:29:31.861 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:40.287 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:40.279+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:29:40.289 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:40.279+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:29:40.291 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:40.288+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:29:40.291 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:40.288+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:29:40.292 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:40.288+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:29:51.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:29:52.352 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:29:52.353 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:29:52.373 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2858,"stamp":"2023-10-21T05:29:52.024034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31067236093,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2517087,"num_read_kb":9796318,"num_write":2467035,"num_write_kb":9865915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89757,"ondisk_log_size":89757,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25002408,"kb_used_data":19732560,"kb_used_omap":4360,"kb_used_meta":5265463,"kb_avail":258113112,"statfs":{"total":289910292480,"available":264307826688,"internally_reserved":0,"allocated":20206141440,"data_stored":39807415714,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4465560,"internal_metadata":5391834216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,8,19,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21569536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":20436,"num_write":5259,"num_write_kb":21036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002276"},"pg_stats":[{"pgid":"2.7","version":"154'8208","reported_seq":11943,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.696994+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:29:49.696994+0000","last_peered":"2023-10-21T05:29:49.696994+0000","last_clean":"2023-10-21T05:29:49.696994+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:29:49.696994+0000","last_undegraded":"2023-10-21T05:29:49.696994+0000","last_fullsized":"2023-10-21T05:29:49.696994+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'4199","reported_seq":8114,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.696678+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:29:49.696678+0000","last_peered":"2023-10-21T05:29:49.696678+0000","last_clean":"2023-10-21T05:29:49.696678+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:29:49.696678+0000","last_undegraded":"2023-10-21T05:29:49.696678+0000","last_fullsized":"2023-10-21T05:29:49.696678+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'12254","reported_seq":17710,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.696890+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:29:49.696890+0000","last_peered":"2023-10-21T05:29:49.696890+0000","last_clean":"2023-10-21T05:29:49.696890+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:29:49.696890+0000","last_undegraded":"2023-10-21T05:29:49.696890+0000","last_fullsized":"2023-10-21T05:29:49.696890+0000","mapping_epoch":11,"log_start":"68'2200","ondisk_log_start":"68'2200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'8252","reported_seq":13001,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.697242+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:29:49.697242+0000","last_peered":"2023-10-21T05:29:49.697242+0000","last_clean":"2023-10-21T05:29:49.697242+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:29:49.697242+0000","last_undegraded":"2023-10-21T05:29:49.697242+0000","last_fullsized":"2023-10-21T05:29:49.697242+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8252,"log_dups_size":0,"ondisk_log_size":8252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'620295","reported_seq":1226518,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:48.820494+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:29:48.820494+0000","last_peered":"2023-10-21T05:29:48.820494+0000","last_clean":"2023-10-21T05:29:48.820494+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:29:48.820494+0000","last_undegraded":"2023-10-21T05:29:48.820494+0000","last_fullsized":"2023-10-21T05:29:48.820494+0000","mapping_epoch":14,"log_start":"153'610262","ondisk_log_start":"153'610262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7683178496,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":610515,"num_read_kb":2413428,"num_write":610140,"num_write_kb":2440560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"154'16635","reported_seq":24042,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:48.100297+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:29:48.100297+0000","last_peered":"2023-10-21T05:29:48.100297+0000","last_clean":"2023-10-21T05:29:48.100297+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:29:48.100297+0000","last_undegraded":"2023-10-21T05:29:48.100297+0000","last_fullsized":"2023-10-21T05:29:48.100297+0000","mapping_epoch":11,"log_start":"103'6600","ondisk_log_start":"103'6600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'627533","reported_seq":1242277,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:50.232288+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:29:50.232288+0000","last_peered":"2023-10-21T05:29:50.232288+0000","last_clean":"2023-10-21T05:29:50.232288+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:29:50.232288+0000","last_undegraded":"2023-10-21T05:29:50.232288+0000","last_fullsized":"2023-10-21T05:29:50.232288+0000","mapping_epoch":14,"log_start":"153'617461","ondisk_log_start":"153'617461","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7778041856,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":619075,"num_read_kb":2446800,"num_write":617292,"num_write_kb":2469168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3872,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.698565+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:29:49.698565+0000","last_peered":"2023-10-21T05:29:49.698565+0000","last_clean":"2023-10-21T05:29:49.698565+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:29:49.698565+0000","last_undegraded":"2023-10-21T05:29:49.698565+0000","last_fullsized":"2023-10-21T05:29:49.698565+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'655720","reported_seq":1294515,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:48.809569+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:29:48.809569+0000","last_peered":"2023-10-21T05:29:48.809569+0000","last_clean":"2023-10-21T05:29:48.809569+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:29:48.809569+0000","last_undegraded":"2023-10-21T05:29:48.809569+0000","last_fullsized":"2023-10-21T05:29:48.809569+0000","mapping_epoch":14,"log_start":"153'645654","ondisk_log_start":"153'645654","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8124452864,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":643314,"num_read_kb":2543464,"num_write":645006,"num_write_kb":2580024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7689,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.698652+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:29:49.698652+0000","last_peered":"2023-10-21T05:29:49.698652+0000","last_clean":"2023-10-21T05:29:49.698652+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:29:49.698652+0000","last_undegraded":"2023-10-21T05:29:49.698652+0000","last_fullsized":"2023-10-21T05:29:49.698652+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'602105","reported_seq":1192825,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:50.241409+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:29:50.241409+0000","last_peered":"2023-10-21T05:29:50.241409+0000","last_clean":"2023-10-21T05:29:50.241409+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:29:50.241409+0000","last_undegraded":"2023-10-21T05:29:50.241409+0000","last_fullsized":"2023-10-21T05:29:50.241409+0000","mapping_epoch":14,"log_start":"153'592060","ondisk_log_start":"153'592060","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4653,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 718s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7480967187,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":594724,"num_read_kb":2351556,"num_write":593439,"num_write_kb":2373750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'8165","reported_seq":11831,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:29:49.697640+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:29:49.697640+0000","last_peered":"2023-10-21T05:29:49.697640+0000","last_clean":"2023-10-21T05:29:49.697640+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:29:49.697640+0000","last_undegraded":"2023-10-21T05:29:49.697640+0000","last_fullsized":"2023-10-21T05:29:49.697640+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":336,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-21T05:28:29.462110+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:28:29.462110+0000","last_peered":"2023-10-21T05:28:29.462110+0000","last_clean":"2023-10-21T05:28:29.462110+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:28:29.462110+0000","last_undegraded":"2023-10-21T05:28:29.462110+0000","last_fullsized":"2023-10-21T05:28:29.462110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31066640403,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2467628,"num_read_kb":9755248,"num_write":2465877,"num_write_kb":9863502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20201545728,"data_stored":39805362214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40216,"ondisk_log_size":40216,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49442,"ondisk_log_size":49442,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739453,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11857132,"kb_used_data":9866084,"kb_used_omap":1521,"kb_used_meta":1989518,"kb_avail":82514708,"statfs":{"total":96636764160,"available":84495060992,"internally_reserved":0,"allocated":10102870016,"data_stored":19903734881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1557939,"internal_metadata":2037267021},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.91500000000000004,"15min":0.998},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.186},"max":{"1min":2.754,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.89300000000000002,"15min":1.026},"min":{"1min":0.27200000000000002,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":2.8969999999999998,"5min":3.3900000000000001,"15min":27.251000000000001},"last":0.873}]},{"osd":1,"last update":"Sat Oct 21 05:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.88200000000000001,"15min":0.89900000000000002},"min":{"1min":0.26700000000000002,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":2.508,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.93100000000000005,"15min":0.98599999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":2.2480000000000002,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.95699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739451,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6677504,"kb_used_data":4844088,"kb_used_omap":1339,"kb_used_meta":1832068,"kb_avail":87694336,"statfs":{"total":96636764160,"available":89799000064,"internally_reserved":0,"allocated":4960346112,"data_stored":9771377717,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1876037980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,14,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.81100000000000005,"15min":0.73499999999999999},"min":{"1min":0.193,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.7809999999999999,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":1.6799999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89000000000000001,"15min":0.83999999999999997},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.8069999999999999,"5min":6.968,"15min":6.968},"last":2.2240000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.99099999999999999,"15min":0.89000000000000001},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.742,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":3.6080000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.93899999999999995,"15min":0.89200000000000002},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.22},"max":{"1min":1.621,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":3.5680000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739451,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6467772,"kb_used_data":5022388,"kb_used_omap":1499,"kb_used_meta":1443876,"kb_avail":87904068,"statfs":{"total":96636764160,"available":90013765632,"internally_reserved":0,"allocated":5142925312,"data_stored":10132303116,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1478529215},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.76600000000000001,"15min":0.77800000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":4.1260000000000003,"5min":4.8090000000000002,"15min":5.923},"last":0.747},{"interface":"front","average":{"1min":0.877,"5min":0.79700000000000004,"15min":0.82799999999999996},"min":{"1min":0.28199999999999997,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":4.1600000000000001,"5min":4.8639999999999999,"15min":6.4050000000000002},"last":0.78700000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.94999999999999996,"15min":1.01},"min":{"1min":0.221,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":3.919,"5min":4.867,"15min":14.053000000000001},"last":6.625},{"interface":"front","average":{"1min":1.399,"5min":1.024,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":9.4469999999999992,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":2.6899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5141086208,"data_stored":10131132416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4959170560,"data_stored":9771032595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10101288960,"data_stored":19903197203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:29:52.375 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:29:52.376 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:58.218 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.216+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:29:58.219 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.216+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:29:58.233 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.228+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:29:58.234 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.228+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:29:58.234 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.228+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:29:58.240 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.234+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:29:58.240 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.234+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:29:58.241 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.234+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:29:58.241 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:29:58.235+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:30:00.590 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:30:00.576+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:30:04.562 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.555+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:30:04.583 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:30:04.583 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:30:04.584 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:30:04.584 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:30:04.584 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:30:04.584 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:30:04.585 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:04.579+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:30:12.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:30:12.585 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:30:12.583+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:30:12.922 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:30:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:30:12.942 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2869,"stamp":"2023-10-21T05:30:12.027404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31101830909,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2525615,"num_read_kb":9830350,"num_write":2475481,"num_write_kb":9899699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89830,"ondisk_log_size":89830,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25315872,"kb_used_data":19799232,"kb_used_omap":4401,"kb_used_meta":5512206,"kb_avail":257799648,"statfs":{"total":289910292480,"available":263986839552,"internally_reserved":0,"allocated":20274413568,"data_stored":39875675542,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4507525,"internal_metadata":5644499067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,14,20,7,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21856256,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5286,"num_read_kb":21096,"num_write":5341,"num_write_kb":21364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002011"},"pg_stats":[{"pgid":"2.7","version":"155'8226","reported_seq":11970,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.178000+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:30:10.178000+0000","last_peered":"2023-10-21T05:30:10.178000+0000","last_clean":"2023-10-21T05:30:10.178000+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:30:10.178000+0000","last_undegraded":"2023-10-21T05:30:10.178000+0000","last_fullsized":"2023-10-21T05:30:10.178000+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'4208","reported_seq":8128,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:09.736696+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:30:09.736696+0000","last_peered":"2023-10-21T05:30:09.736696+0000","last_clean":"2023-10-21T05:30:09.736696+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:30:09.736696+0000","last_undegraded":"2023-10-21T05:30:09.736696+0000","last_fullsized":"2023-10-21T05:30:09.736696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'12281","reported_seq":17751,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.178884+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:30:10.178884+0000","last_peered":"2023-10-21T05:30:10.178884+0000","last_clean":"2023-10-21T05:30:10.178884+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:30:10.178884+0000","last_undegraded":"2023-10-21T05:30:10.178884+0000","last_fullsized":"2023-10-21T05:30:10.178884+0000","mapping_epoch":11,"log_start":"68'2200","ondisk_log_start":"68'2200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'8270","reported_seq":13029,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.178640+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:30:10.178640+0000","last_peered":"2023-10-21T05:30:10.178640+0000","last_clean":"2023-10-21T05:30:10.178640+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:30:10.178640+0000","last_undegraded":"2023-10-21T05:30:10.178640+0000","last_fullsized":"2023-10-21T05:30:10.178640+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8270,"log_dups_size":0,"ondisk_log_size":8270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'622388","reported_seq":1230702,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:08.889846+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:30:08.889846+0000","last_peered":"2023-10-21T05:30:08.889846+0000","last_clean":"2023-10-21T05:30:08.889846+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:30:08.889846+0000","last_undegraded":"2023-10-21T05:30:08.889846+0000","last_fullsized":"2023-10-21T05:30:08.889846+0000","mapping_epoch":14,"log_start":"153'612362","ondisk_log_start":"153'612362","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7691767808,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":612607,"num_read_kb":2421768,"num_write":612233,"num_write_kb":2448932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'16677","reported_seq":24102,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:08.865755+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:30:08.865755+0000","last_peered":"2023-10-21T05:30:08.865755+0000","last_clean":"2023-10-21T05:30:08.865755+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:30:08.865755+0000","last_undegraded":"2023-10-21T05:30:08.865755+0000","last_fullsized":"2023-10-21T05:30:08.865755+0000","mapping_epoch":11,"log_start":"103'6600","ondisk_log_start":"103'6600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'629591","reported_seq":1246420,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.410002+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:30:10.410002+0000","last_peered":"2023-10-21T05:30:10.410002+0000","last_clean":"2023-10-21T05:30:10.410002+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:30:10.410002+0000","last_undegraded":"2023-10-21T05:30:10.410002+0000","last_fullsized":"2023-10-21T05:30:10.410002+0000","mapping_epoch":14,"log_start":"153'619561","ondisk_log_start":"153'619561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7786463232,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":621158,"num_read_kb":2455116,"num_write":619350,"num_write_kb":2477400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3881,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.181185+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:30:10.181185+0000","last_peered":"2023-10-21T05:30:10.181185+0000","last_clean":"2023-10-21T05:30:10.181185+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:30:10.181185+0000","last_undegraded":"2023-10-21T05:30:10.181185+0000","last_fullsized":"2023-10-21T05:30:10.181185+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'657898","reported_seq":1298933,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:08.893019+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:30:08.893019+0000","last_peered":"2023-10-21T05:30:08.893019+0000","last_clean":"2023-10-21T05:30:08.893019+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:30:08.893019+0000","last_undegraded":"2023-10-21T05:30:08.893019+0000","last_fullsized":"2023-10-21T05:30:08.893019+0000","mapping_epoch":14,"log_start":"153'647854","ondisk_log_start":"153'647854","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8133365760,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":645547,"num_read_kb":2552380,"num_write":647184,"num_write_kb":2588736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7706,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.180165+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:30:10.180165+0000","last_peered":"2023-10-21T05:30:10.180165+0000","last_clean":"2023-10-21T05:30:10.180165+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:30:10.180165+0000","last_undegraded":"2023-10-21T05:30:10.180165+0000","last_fullsized":"2023-10-21T05:30:10.180165+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'604317","reported_seq":1197165,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:30:10.415804+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:30:10.415804+0000","last_peered":"2023-10-21T05:30:10.415804+0000","last_clean":"2023-10-21T05:30:10.415804+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:30:10.415804+0000","last_undegraded":"2023-10-21T05:30:10.415804+0000","last_fullsized":"2023-10-21T05:30:10.415804+0000","mapping_epoch":14,"log_start":"153'594260","ondisk_log_start":"153'594260","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4767,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 738s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7489638419,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":596844,"num_read_kb":2360016,"num_write":595556,"num_write_kb":2382218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'8183","reported_seq":11859,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:10.178402+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:30:10.178402+0000","last_peered":"2023-10-21T05:30:10.178402+0000","last_clean":"2023-10-21T05:30:10.178402+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:30:10.178402+0000","last_undegraded":"2023-10-21T05:30:10.178402+0000","last_fullsized":"2023-10-21T05:30:10.178402+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8183,"log_dups_size":0,"ondisk_log_size":8183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":337,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-21T05:28:49.629409+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:28:49.629409+0000","last_peered":"2023-10-21T05:28:49.629409+0000","last_clean":"2023-10-21T05:28:49.629409+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:28:49.629409+0000","last_undegraded":"2023-10-21T05:28:49.629409+0000","last_fullsized":"2023-10-21T05:28:49.629409+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31101235219,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2476156,"num_read_kb":9789280,"num_write":2474323,"num_write_kb":9897286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20270616576,"data_stored":39874433062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40157,"ondisk_log_size":40157,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49574,"ondisk_log_size":49574,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739457,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11981160,"kb_used_data":9899804,"kb_used_omap":1562,"kb_used_meta":2079781,"kb_avail":82390680,"statfs":{"total":96636764160,"available":84368056320,"internally_reserved":0,"allocated":10137399296,"data_stored":19938260061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":2129696352},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.91500000000000004,"15min":0.998},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.186},"max":{"1min":2.754,"5min":8.3439999999999994,"15min":27.184999999999999},"last":3.7749999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.89300000000000002,"15min":1.026},"min":{"1min":0.27200000000000002,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":2.8969999999999998,"5min":3.3900000000000001,"15min":27.251000000000001},"last":3.681}]},{"osd":1,"last update":"Sat Oct 21 05:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.88200000000000001,"15min":0.89900000000000002},"min":{"1min":0.26700000000000002,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":2.508,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.309},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.93100000000000005,"15min":0.98599999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":2.2480000000000002,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":3.7330000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739455,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6778452,"kb_used_data":4860748,"kb_used_omap":1339,"kb_used_meta":1916356,"kb_avail":87593388,"statfs":{"total":96636764160,"available":89695629312,"internally_reserved":0,"allocated":4977405952,"data_stored":9788433457,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":1962348892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,8,11,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.81100000000000005,"15min":0.73499999999999999},"min":{"1min":0.193,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.7809999999999999,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":1.4450000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89000000000000001,"15min":0.83999999999999997},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.8069999999999999,"5min":6.968,"15min":6.968},"last":1.48}]},{"osd":2,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.99099999999999999,"15min":0.89000000000000001},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.742,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.93899999999999995,"15min":0.89200000000000002},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.22},"max":{"1min":1.621,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6556260,"kb_used_data":5038680,"kb_used_omap":1499,"kb_used_meta":1516068,"kb_avail":87815580,"statfs":{"total":96636764160,"available":89923153920,"internally_reserved":0,"allocated":5159608320,"data_stored":10148982024,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1552453823},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.76600000000000001,"15min":0.77800000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":4.1260000000000003,"5min":4.8090000000000002,"15min":5.923},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.877,"5min":0.79700000000000004,"15min":0.82799999999999996},"min":{"1min":0.28199999999999997,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":4.1600000000000001,"5min":4.8639999999999999,"15min":6.4050000000000002},"last":0.86699999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.94999999999999996,"15min":1.01},"min":{"1min":0.221,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":3.919,"5min":4.867,"15min":14.053000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.399,"5min":1.024,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":9.4469999999999992,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5158584320,"data_stored":10148630528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4976250880,"data_stored":9788112915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10135781376,"data_stored":19937689619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:30:12.944 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:30:12.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:30:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:30:13.535 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:30:15.499 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:15.498+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:30:15.500 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:15.498+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:30:15.500 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:15.498+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:30:15.500 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:15.498+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:30:15.501 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:30:15.498+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:30:31.324 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:30:31.322+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:30:31.325 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:30:31.322+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:30:33.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:30:34.088 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:30:34.089 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:30:34.119 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2880,"stamp":"2023-10-21T05:30:34.033683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31136909053,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2534042,"num_read_kb":9863982,"num_write":2484051,"num_write_kb":9933979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89866,"ondisk_log_size":89866,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24730520,"kb_used_data":19867712,"kb_used_omap":4401,"kb_used_meta":4858382,"kb_avail":258385000,"statfs":{"total":289910292480,"available":264586240000,"internally_reserved":0,"allocated":20344537088,"data_stored":39945799062,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4507525,"internal_metadata":4974983291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,4,15,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17637376,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4273,"num_read_kb":17060,"num_write":4308,"num_write_kb":17232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004506"},"pg_stats":[{"pgid":"2.7","version":"155'8254","reported_seq":12004,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:27.202576+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:30:27.202576+0000","last_peered":"2023-10-21T05:30:27.202576+0000","last_clean":"2023-10-21T05:30:27.202576+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:30:27.202576+0000","last_undegraded":"2023-10-21T05:30:27.202576+0000","last_fullsized":"2023-10-21T05:30:27.202576+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8254,"log_dups_size":0,"ondisk_log_size":8254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'4222","reported_seq":8145,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:26.232570+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:30:26.232570+0000","last_peered":"2023-10-21T05:30:26.232570+0000","last_clean":"2023-10-21T05:30:26.232570+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:30:26.232570+0000","last_undegraded":"2023-10-21T05:30:26.232570+0000","last_fullsized":"2023-10-21T05:30:26.232570+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'12323","reported_seq":17802,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:27.213964+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:30:27.213964+0000","last_peered":"2023-10-21T05:30:27.213964+0000","last_clean":"2023-10-21T05:30:27.213964+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:30:27.213964+0000","last_undegraded":"2023-10-21T05:30:27.213964+0000","last_fullsized":"2023-10-21T05:30:27.213964+0000","mapping_epoch":11,"log_start":"68'2300","ondisk_log_start":"68'2300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'8298","reported_seq":13063,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:27.197902+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:30:27.197902+0000","last_peered":"2023-10-21T05:30:27.197902+0000","last_clean":"2023-10-21T05:30:27.197902+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:30:27.197902+0000","last_undegraded":"2023-10-21T05:30:27.197902+0000","last_fullsized":"2023-10-21T05:30:27.197902+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'624527","reported_seq":1234924,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:29.136168+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:30:29.136168+0000","last_peered":"2023-10-21T05:30:29.136168+0000","last_clean":"2023-10-21T05:30:29.136168+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:30:29.136168+0000","last_undegraded":"2023-10-21T05:30:29.136168+0000","last_fullsized":"2023-10-21T05:30:29.136168+0000","mapping_epoch":14,"log_start":"154'614462","ondisk_log_start":"154'614462","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7700541440,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":614691,"num_read_kb":2430084,"num_write":614372,"num_write_kb":2457488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'16733","reported_seq":24170,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:28.695440+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:30:28.695440+0000","last_peered":"2023-10-21T05:30:28.695440+0000","last_clean":"2023-10-21T05:30:28.695440+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:30:28.695440+0000","last_undegraded":"2023-10-21T05:30:28.695440+0000","last_fullsized":"2023-10-21T05:30:28.695440+0000","mapping_epoch":11,"log_start":"103'6700","ondisk_log_start":"103'6700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'631754","reported_seq":1250695,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:30.555484+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:30:30.555484+0000","last_peered":"2023-10-21T05:30:30.555484+0000","last_clean":"2023-10-21T05:30:30.555484+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:30:30.555484+0000","last_undegraded":"2023-10-21T05:30:30.555484+0000","last_fullsized":"2023-10-21T05:30:30.555484+0000","mapping_epoch":14,"log_start":"154'621661","ondisk_log_start":"154'621661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7795302400,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":623270,"num_read_kb":2463544,"num_write":621513,"num_write_kb":2486052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3887,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:25.537704+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:30:25.537704+0000","last_peered":"2023-10-21T05:30:25.537704+0000","last_clean":"2023-10-21T05:30:25.537704+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:30:25.537704+0000","last_undegraded":"2023-10-21T05:30:25.537704+0000","last_fullsized":"2023-10-21T05:30:25.537704+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'660174","reported_seq":1303378,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:29.131705+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:30:29.131705+0000","last_peered":"2023-10-21T05:30:29.131705+0000","last_clean":"2023-10-21T05:30:29.131705+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:30:29.131705+0000","last_undegraded":"2023-10-21T05:30:29.131705+0000","last_fullsized":"2023-10-21T05:30:29.131705+0000","mapping_epoch":14,"log_start":"154'650154","ondisk_log_start":"154'650154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8142675968,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":647718,"num_read_kb":2561048,"num_write":649460,"num_write_kb":2597840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7719,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:25.537234+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:30:25.537234+0000","last_peered":"2023-10-21T05:30:25.537234+0000","last_clean":"2023-10-21T05:30:25.537234+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:30:25.537234+0000","last_undegraded":"2023-10-21T05:30:25.537234+0000","last_fullsized":"2023-10-21T05:30:25.537234+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'606379","reported_seq":1201300,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:30:30.556431+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:30:30.556431+0000","last_peered":"2023-10-21T05:30:30.556431+0000","last_clean":"2023-10-21T05:30:30.556431+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:30:30.556431+0000","last_undegraded":"2023-10-21T05:30:30.556431+0000","last_fullsized":"2023-10-21T05:30:30.556431+0000","mapping_epoch":14,"log_start":"154'596360","ondisk_log_start":"154'596360","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4850,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 759s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7497793555,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":598904,"num_read_kb":2368236,"num_write":597548,"num_write_kb":2390186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'8211","reported_seq":11893,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:27.214826+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:30:27.214826+0000","last_peered":"2023-10-21T05:30:27.214826+0000","last_clean":"2023-10-21T05:30:27.214826+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:30:27.214826+0000","last_undegraded":"2023-10-21T05:30:27.214826+0000","last_fullsized":"2023-10-21T05:30:27.214826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8211,"log_dups_size":0,"ondisk_log_size":8211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":338,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.646442+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:29:59.646442+0000","last_peered":"2023-10-21T05:29:59.646442+0000","last_clean":"2023-10-21T05:29:59.646442+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:29:59.646442+0000","last_undegraded":"2023-10-21T05:29:59.646442+0000","last_fullsized":"2023-10-21T05:29:59.646442+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31136313363,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2484583,"num_read_kb":9822912,"num_write":2482893,"num_write_kb":9931566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20340682752,"data_stored":39944499238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49570,"ondisk_log_size":49570,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739461,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11161404,"kb_used_data":9932404,"kb_used_omap":1562,"kb_used_meta":1227429,"kb_avail":83210436,"statfs":{"total":96636764160,"available":85207486464,"internally_reserved":0,"allocated":10170781696,"data_stored":19971642461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":1256887904},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.91500000000000004,"15min":0.998},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.186},"max":{"1min":2.754,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.89300000000000002,"15min":1.026},"min":{"1min":0.27200000000000002,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":2.8969999999999998,"5min":3.3900000000000001,"15min":27.251000000000001},"last":0.63}]},{"osd":1,"last update":"Sat Oct 21 05:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.88200000000000001,"15min":0.89900000000000002},"min":{"1min":0.26700000000000002,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":2.508,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.93100000000000005,"15min":0.98599999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":2.2480000000000002,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.25700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739459,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6886140,"kb_used_data":4878196,"kb_used_omap":1339,"kb_used_meta":2006596,"kb_avail":87485700,"statfs":{"total":96636764160,"available":89585356800,"internally_reserved":0,"allocated":4995272704,"data_stored":9806300209,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1371812,"internal_metadata":2054754652},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.81100000000000005,"15min":0.73499999999999999},"min":{"1min":0.193,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.7809999999999999,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89000000000000001,"15min":0.83999999999999997},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.8069999999999999,"5min":6.968,"15min":6.968},"last":0.247}]},{"osd":2,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.99099999999999999,"15min":0.89000000000000001},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.742,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.93899999999999995,"15min":0.89200000000000002},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.22},"max":{"1min":1.621,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6682976,"kb_used_data":5057112,"kb_used_omap":1499,"kb_used_meta":1624356,"kb_avail":87688864,"statfs":{"total":96636764160,"available":89793396736,"internally_reserved":0,"allocated":5178482688,"data_stored":10167856392,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1663340735},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.76600000000000001,"15min":0.77800000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":4.1260000000000003,"5min":4.8090000000000002,"15min":5.923},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.79700000000000004,"15min":0.82799999999999996},"min":{"1min":0.28199999999999997,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":4.1600000000000001,"5min":4.8639999999999999,"15min":6.4050000000000002},"last":0.85999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.94999999999999996,"15min":1.01},"min":{"1min":0.221,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":3.919,"5min":4.867,"15min":14.053000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.399,"5min":1.024,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":9.4469999999999992,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5176664064,"data_stored":10166710272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4993269760,"data_stored":9805131795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10170748928,"data_stored":19972657171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:30:34.121 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:30:34.121 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:30:54.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:30:54.654 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:30:54.655 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:30:54.679 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2890,"stamp":"2023-10-21T05:30:54.037139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31171766013,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2542337,"num_read_kb":9897106,"num_write":2492567,"num_write_kb":9968043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89967,"ondisk_log_size":89967,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24535004,"kb_used_data":19935236,"kb_used_omap":4483,"kb_used_meta":4595260,"kb_avail":258580516,"statfs":{"total":289910292480,"available":264786448384,"internally_reserved":0,"allocated":20413681664,"data_stored":40014943638,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4591586,"internal_metadata":4705546270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,6,7,11,40,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17305600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4077,"num_read_kb":16276,"num_write":4228,"num_write_kb":16912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002112"},"pg_stats":[{"pgid":"2.7","version":"155'8266","reported_seq":12024,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:48.342325+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:30:48.342325+0000","last_peered":"2023-10-21T05:30:48.342325+0000","last_clean":"2023-10-21T05:30:48.342325+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:30:48.342325+0000","last_undegraded":"2023-10-21T05:30:48.342325+0000","last_fullsized":"2023-10-21T05:30:48.342325+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8266,"log_dups_size":0,"ondisk_log_size":8266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'4228","reported_seq":8155,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:46.016661+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:30:46.016661+0000","last_peered":"2023-10-21T05:30:46.016661+0000","last_clean":"2023-10-21T05:30:46.016661+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:30:46.016661+0000","last_undegraded":"2023-10-21T05:30:46.016661+0000","last_fullsized":"2023-10-21T05:30:46.016661+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'12341","reported_seq":17832,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:48.344489+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:30:48.344489+0000","last_peered":"2023-10-21T05:30:48.344489+0000","last_clean":"2023-10-21T05:30:48.344489+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:30:48.344489+0000","last_undegraded":"2023-10-21T05:30:48.344489+0000","last_fullsized":"2023-10-21T05:30:48.344489+0000","mapping_epoch":11,"log_start":"68'2300","ondisk_log_start":"68'2300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'8310","reported_seq":13083,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:48.342711+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:30:48.342711+0000","last_peered":"2023-10-21T05:30:48.342711+0000","last_clean":"2023-10-21T05:30:48.342711+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:30:48.342711+0000","last_undegraded":"2023-10-21T05:30:48.342711+0000","last_fullsized":"2023-10-21T05:30:48.342711+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8310,"log_dups_size":0,"ondisk_log_size":8310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'626619","reported_seq":1239078,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:49.301159+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:30:49.301159+0000","last_peered":"2023-10-21T05:30:49.301159+0000","last_clean":"2023-10-21T05:30:49.301159+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:30:49.301159+0000","last_undegraded":"2023-10-21T05:30:49.301159+0000","last_fullsized":"2023-10-21T05:30:49.301159+0000","mapping_epoch":14,"log_start":"154'616562","ondisk_log_start":"154'616562","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7709102080,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":616750,"num_read_kb":2438312,"num_write":616464,"num_write_kb":2465856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'16770","reported_seq":24223,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:49.276854+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:30:49.276854+0000","last_peered":"2023-10-21T05:30:49.276854+0000","last_clean":"2023-10-21T05:30:49.276854+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:30:49.276854+0000","last_undegraded":"2023-10-21T05:30:49.276854+0000","last_fullsized":"2023-10-21T05:30:49.276854+0000","mapping_epoch":11,"log_start":"103'6700","ondisk_log_start":"103'6700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'633834","reported_seq":1254839,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:50.688083+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:30:50.688083+0000","last_peered":"2023-10-21T05:30:50.688083+0000","last_clean":"2023-10-21T05:30:50.688083+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:30:50.688083+0000","last_undegraded":"2023-10-21T05:30:50.688083+0000","last_fullsized":"2023-10-21T05:30:50.688083+0000","mapping_epoch":14,"log_start":"154'623761","ondisk_log_start":"154'623761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7803813888,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":625335,"num_read_kb":2471792,"num_write":623593,"num_write_kb":2494372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3895,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:46.017656+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:30:46.017656+0000","last_peered":"2023-10-21T05:30:46.017656+0000","last_clean":"2023-10-21T05:30:46.017656+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:30:46.017656+0000","last_undegraded":"2023-10-21T05:30:46.017656+0000","last_fullsized":"2023-10-21T05:30:46.017656+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'662422","reported_seq":1307864,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:49.310948+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:30:49.310948+0000","last_peered":"2023-10-21T05:30:49.310948+0000","last_clean":"2023-10-21T05:30:49.310948+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:30:49.310948+0000","last_undegraded":"2023-10-21T05:30:49.310948+0000","last_fullsized":"2023-10-21T05:30:49.310948+0000","mapping_epoch":14,"log_start":"154'652354","ondisk_log_start":"154'652354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8151883776,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":649959,"num_read_kb":2569992,"num_write":651708,"num_write_kb":2606832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7735,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:46.017937+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:30:46.017937+0000","last_peered":"2023-10-21T05:30:46.017937+0000","last_clean":"2023-10-21T05:30:46.017937+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:30:46.017937+0000","last_undegraded":"2023-10-21T05:30:46.017937+0000","last_fullsized":"2023-10-21T05:30:46.017937+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'608563","reported_seq":1205421,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:30:50.707763+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:30:50.707763+0000","last_peered":"2023-10-21T05:30:50.707763+0000","last_clean":"2023-10-21T05:30:50.707763+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:30:50.707763+0000","last_undegraded":"2023-10-21T05:30:50.707763+0000","last_fullsized":"2023-10-21T05:30:50.707763+0000","mapping_epoch":14,"log_start":"154'598560","ondisk_log_start":"154'598560","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":4959,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 779s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7506370579,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":600834,"num_read_kb":2375940,"num_write":599644,"num_write_kb":2398570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'8223","reported_seq":11913,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:48.350929+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:30:48.350929+0000","last_peered":"2023-10-21T05:30:48.350929+0000","last_clean":"2023-10-21T05:30:48.350929+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:30:48.350929+0000","last_undegraded":"2023-10-21T05:30:48.350929+0000","last_fullsized":"2023-10-21T05:30:48.350929+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8223,"log_dups_size":0,"ondisk_log_size":8223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":338,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.646442+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:29:59.646442+0000","last_peered":"2023-10-21T05:29:59.646442+0000","last_clean":"2023-10-21T05:29:59.646442+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:29:59.646442+0000","last_undegraded":"2023-10-21T05:29:59.646442+0000","last_fullsized":"2023-10-21T05:29:59.646442+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31171170323,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2492878,"num_read_kb":9856036,"num_write":2491409,"num_write_kb":9965630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20410232832,"data_stored":40014049318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40201,"ondisk_log_size":40201,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49667,"ondisk_log_size":49667,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739465,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11411720,"kb_used_data":9966208,"kb_used_omap":1562,"kb_used_meta":1443941,"kb_avail":82960120,"statfs":{"total":96636764160,"available":84951162880,"internally_reserved":0,"allocated":10205396992,"data_stored":20006257757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":1478596192},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.86899999999999999,"15min":0.997},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.186},"max":{"1min":3.7749999999999999,"5min":8.3439999999999994,"15min":27.184999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.86699999999999999,"15min":1.0109999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.681,"5min":3.681,"15min":27.251000000000001},"last":1.6220000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.82199999999999995,"15min":0.86199999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":2.2730000000000001,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.04,"15min":1.006},"min":{"1min":0.25700000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":3.7330000000000001,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":1.653}]}]},{"osd":1,"up_from":8,"seq":34359739463,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6350876,"kb_used_data":4894676,"kb_used_omap":1421,"kb_used_meta":1454770,"kb_avail":88020964,"statfs":{"total":96636764160,"available":90133467136,"internally_reserved":0,"allocated":5012148224,"data_stored":9823175729,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1489684735},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,3,32,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:30:34 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":0.90800000000000003,"15min":0.77900000000000003},"min":{"1min":0.223,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.6269999999999998,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.94599999999999995,"15min":0.85599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":4.6609999999999996,"5min":6.968,"15min":6.968},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:30:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.0960000000000001,"15min":0.93300000000000005},"min":{"1min":0.27000000000000002,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":5.3289999999999997,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.435},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.0600000000000001,"15min":0.93600000000000005},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.22},"max":{"1min":5.2880000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.47399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739463,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6772408,"kb_used_data":5074352,"kb_used_omap":1499,"kb_used_meta":1696548,"kb_avail":87599432,"statfs":{"total":96636764160,"available":89701818368,"internally_reserved":0,"allocated":5196136448,"data_stored":10185510152,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1737265343},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80300000000000005,"15min":0.77600000000000002},"min":{"1min":0.21199999999999999,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":3.0670000000000002,"5min":4.8090000000000002,"15min":5.923},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.871,"15min":0.82599999999999996},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":3.1019999999999999,"5min":4.8639999999999999,"15min":5.968},"last":0.318}]},{"osd":2,"last update":"Sat Oct 21 05:30:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.0600000000000001,"15min":1.042},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.1210000000000004,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.1160000000000001,"15min":1.0589999999999999},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.0350000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":1.8460000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5194432512,"data_stored":10184478720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5010391040,"data_stored":9822253075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10205409280,"data_stored":20007317523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:30:54.680 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:30:54.681 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:31:12.686 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:12.684+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:31:14.682 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:31:15.201 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:31:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:31:15.225 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2901,"stamp":"2023-10-21T05:31:14.041046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31207548669,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2550690,"num_read_kb":9930438,"num_write":2501314,"num_write_kb":10003031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90015,"ondisk_log_size":90015,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24906636,"kb_used_data":20006060,"kb_used_omap":4483,"kb_used_meta":4896060,"kb_avail":258208884,"statfs":{"total":289910292480,"available":264405897216,"internally_reserved":0,"allocated":20486205440,"data_stored":40087459128,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4591586,"internal_metadata":5013565470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,5,7,12,22,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18108416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4251,"num_read_kb":16956,"num_write":4429,"num_write_kb":17716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001655"},"pg_stats":[{"pgid":"2.7","version":"155'8282","reported_seq":12049,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.248593+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:31:09.248593+0000","last_peered":"2023-10-21T05:31:09.248593+0000","last_clean":"2023-10-21T05:31:09.248593+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:31:09.248593+0000","last_undegraded":"2023-10-21T05:31:09.248593+0000","last_fullsized":"2023-10-21T05:31:09.248593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'4236","reported_seq":8168,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.231804+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:31:09.231804+0000","last_peered":"2023-10-21T05:31:09.231804+0000","last_clean":"2023-10-21T05:31:09.231804+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:31:09.231804+0000","last_undegraded":"2023-10-21T05:31:09.231804+0000","last_fullsized":"2023-10-21T05:31:09.231804+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'12365","reported_seq":17869,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.249360+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:31:09.249360+0000","last_peered":"2023-10-21T05:31:09.249360+0000","last_clean":"2023-10-21T05:31:09.249360+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:31:09.249360+0000","last_undegraded":"2023-10-21T05:31:09.249360+0000","last_fullsized":"2023-10-21T05:31:09.249360+0000","mapping_epoch":11,"log_start":"68'2300","ondisk_log_start":"68'2300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'8326","reported_seq":13108,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.243814+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:31:09.243814+0000","last_peered":"2023-10-21T05:31:09.243814+0000","last_clean":"2023-10-21T05:31:09.243814+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:31:09.243814+0000","last_undegraded":"2023-10-21T05:31:09.243814+0000","last_fullsized":"2023-10-21T05:31:09.243814+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'628761","reported_seq":1243193,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.406792+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:31:09.406792+0000","last_peered":"2023-10-21T05:31:09.406792+0000","last_clean":"2023-10-21T05:31:09.406792+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:31:09.406792+0000","last_undegraded":"2023-10-21T05:31:09.406792+0000","last_fullsized":"2023-10-21T05:31:09.406792+0000","mapping_epoch":14,"log_start":"154'618662","ondisk_log_start":"154'618662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7717859328,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":618724,"num_read_kb":2446196,"num_write":618606,"num_write_kb":2474424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'16819","reported_seq":24288,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:08.472502+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:31:08.472502+0000","last_peered":"2023-10-21T05:31:08.472502+0000","last_clean":"2023-10-21T05:31:08.472502+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:31:08.472502+0000","last_undegraded":"2023-10-21T05:31:08.472502+0000","last_fullsized":"2023-10-21T05:31:08.472502+0000","mapping_epoch":11,"log_start":"103'6800","ondisk_log_start":"103'6800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'636074","reported_seq":1259196,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:10.850086+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:31:10.850086+0000","last_peered":"2023-10-21T05:31:10.850086+0000","last_clean":"2023-10-21T05:31:10.850086+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:31:10.850086+0000","last_undegraded":"2023-10-21T05:31:10.850086+0000","last_fullsized":"2023-10-21T05:31:10.850086+0000","mapping_epoch":14,"log_start":"154'626061","ondisk_log_start":"154'626061","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7812980736,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":627455,"num_read_kb":2480248,"num_write":625833,"num_write_kb":2503332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3904,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:06.501136+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:31:06.501136+0000","last_peered":"2023-10-21T05:31:06.501136+0000","last_clean":"2023-10-21T05:31:06.501136+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:31:06.501136+0000","last_undegraded":"2023-10-21T05:31:06.501136+0000","last_fullsized":"2023-10-21T05:31:06.501136+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'664713","reported_seq":1312400,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.395271+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:31:09.395271+0000","last_peered":"2023-10-21T05:31:09.395271+0000","last_clean":"2023-10-21T05:31:09.395271+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:31:09.395271+0000","last_undegraded":"2023-10-21T05:31:09.395271+0000","last_fullsized":"2023-10-21T05:31:09.395271+0000","mapping_epoch":14,"log_start":"154'654654","ondisk_log_start":"154'654654","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8161259520,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":652200,"num_read_kb":2578928,"num_write":653999,"num_write_kb":2615996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7752,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:06.516483+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:31:06.516483+0000","last_peered":"2023-10-21T05:31:06.516483+0000","last_clean":"2023-10-21T05:31:06.516483+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:31:06.516483+0000","last_undegraded":"2023-10-21T05:31:06.516483+0000","last_fullsized":"2023-10-21T05:31:06.516483+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'610710","reported_seq":1209602,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:31:10.874560+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:31:10.874560+0000","last_peered":"2023-10-21T05:31:10.874560+0000","last_clean":"2023-10-21T05:31:10.874560+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:31:10.874560+0000","last_undegraded":"2023-10-21T05:31:10.874560+0000","last_fullsized":"2023-10-21T05:31:10.874560+0000","mapping_epoch":14,"log_start":"154'600661","ondisk_log_start":"154'600661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5051,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 799s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7514853395,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":602852,"num_read_kb":2383996,"num_write":601718,"num_write_kb":2406866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'8239","reported_seq":11938,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:31:09.261734+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:31:09.261734+0000","last_peered":"2023-10-21T05:31:09.261734+0000","last_clean":"2023-10-21T05:31:09.261734+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:31:09.261734+0000","last_undegraded":"2023-10-21T05:31:09.261734+0000","last_fullsized":"2023-10-21T05:31:09.261734+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8239,"log_dups_size":0,"ondisk_log_size":8239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":339,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:30.556606+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:30:30.556606+0000","last_peered":"2023-10-21T05:30:30.556606+0000","last_clean":"2023-10-21T05:30:30.556606+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:30:30.556606+0000","last_undegraded":"2023-10-21T05:30:30.556606+0000","last_fullsized":"2023-10-21T05:30:30.556606+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31206952979,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2501231,"num_read_kb":9889368,"num_write":2500156,"num_write_kb":10000618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20481736704,"data_stored":40085553190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40220,"ondisk_log_size":40220,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49696,"ondisk_log_size":49696,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739469,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11592220,"kb_used_data":10002320,"kb_used_omap":1562,"kb_used_meta":1588325,"kb_avail":82779620,"statfs":{"total":96636764160,"available":84766330880,"internally_reserved":0,"allocated":10242375680,"data_stored":20043232302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":1626445408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.86899999999999999,"15min":0.997},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.186},"max":{"1min":3.7749999999999999,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.371},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.86699999999999999,"15min":1.0109999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.681,"5min":3.681,"15min":27.251000000000001},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.82199999999999995,"15min":0.86199999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":2.2730000000000001,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.238,"5min":1.04,"15min":1.006},"min":{"1min":0.25700000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":3.7330000000000001,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.55900000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739467,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6440056,"kb_used_data":4911660,"kb_used_omap":1421,"kb_used_meta":1526962,"kb_avail":87931784,"statfs":{"total":96636764160,"available":90042146816,"internally_reserved":0,"allocated":5029539840,"data_stored":9840563202,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1563609343},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:30:34 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":0.90800000000000003,"15min":0.77900000000000003},"min":{"1min":0.223,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.6269999999999998,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.94599999999999995,"15min":0.85599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":4.6609999999999996,"5min":6.968,"15min":6.968},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:30:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.0960000000000001,"15min":0.93300000000000005},"min":{"1min":0.27000000000000002,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":5.3289999999999997,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.0600000000000001,"15min":0.93600000000000005},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.22},"max":{"1min":5.2880000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":1.024}]}]},{"osd":0,"up_from":8,"seq":34359739467,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6874360,"kb_used_data":5092080,"kb_used_omap":1499,"kb_used_meta":1780772,"kb_avail":87497480,"statfs":{"total":96636764160,"available":89597419520,"internally_reserved":0,"allocated":5214289920,"data_stored":10203663624,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1823510719},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80300000000000005,"15min":0.77600000000000002},"min":{"1min":0.21199999999999999,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":3.0670000000000002,"5min":4.8090000000000002,"15min":5.923},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.871,"15min":0.82599999999999996},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":3.1019999999999999,"5min":4.8639999999999999,"15min":5.968},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Oct 21 05:30:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.0600000000000001,"15min":1.042},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.1210000000000004,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.1160000000000001,"15min":1.0589999999999999},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.0350000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5212606464,"data_stored":10202652672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5028048896,"data_stored":9839910931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10241081344,"data_stored":20042989587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:31:15.226 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:31:15.227 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:31:24.087 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:24.083+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:31:24.088 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:24.083+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:31:25.740 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:25.734+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:31:25.740 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:25.734+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:31:30.892 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.887+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:31:30.895 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.893+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:31:30.896 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.893+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:31:30.896 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.893+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:31:30.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.898+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:31:30.900 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.898+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:31:30.901 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.898+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:31:30.901 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:30.898+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:31:32.071 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:32.068+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:31:35.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:31:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:31:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:31:35.793 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2911,"stamp":"2023-10-21T05:31:34.044551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31243073277,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2558854,"num_read_kb":9962994,"num_write":2509991,"num_write_kb":10037739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90157,"ondisk_log_size":90157,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25273468,"kb_used_data":20074916,"kb_used_omap":4483,"kb_used_meta":5194044,"kb_avail":257842052,"statfs":{"total":289910292480,"available":264030261248,"internally_reserved":0,"allocated":20556713984,"data_stored":40157963529,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4591586,"internal_metadata":5318701086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,4,13,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17686528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4003,"num_read_kb":15960,"num_write":4320,"num_write_kb":17280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002065"},"pg_stats":[{"pgid":"2.7","version":"156'8308","reported_seq":12084,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:30.896319+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:31:30.896319+0000","last_peered":"2023-10-21T05:31:30.896319+0000","last_clean":"2023-10-21T05:31:30.896319+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:31:30.896319+0000","last_undegraded":"2023-10-21T05:31:30.896319+0000","last_fullsized":"2023-10-21T05:31:30.896319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'4249","reported_seq":8185,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:30.868457+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:31:30.868457+0000","last_peered":"2023-10-21T05:31:30.868457+0000","last_clean":"2023-10-21T05:31:30.868457+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:31:30.868457+0000","last_undegraded":"2023-10-21T05:31:30.868457+0000","last_fullsized":"2023-10-21T05:31:30.868457+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'12404","reported_seq":17920,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:30.912119+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:31:30.912119+0000","last_peered":"2023-10-21T05:31:30.912119+0000","last_clean":"2023-10-21T05:31:30.912119+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:31:30.912119+0000","last_undegraded":"2023-10-21T05:31:30.912119+0000","last_fullsized":"2023-10-21T05:31:30.912119+0000","mapping_epoch":11,"log_start":"70'2400","ondisk_log_start":"70'2400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'8352","reported_seq":13142,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:30.900005+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:31:30.900005+0000","last_peered":"2023-10-21T05:31:30.900005+0000","last_clean":"2023-10-21T05:31:30.900005+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:31:30.900005+0000","last_undegraded":"2023-10-21T05:31:30.900005+0000","last_fullsized":"2023-10-21T05:31:30.900005+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'630933","reported_seq":1247348,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:29.507015+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:31:29.507015+0000","last_peered":"2023-10-21T05:31:29.507015+0000","last_clean":"2023-10-21T05:31:29.507015+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:31:29.507015+0000","last_undegraded":"2023-10-21T05:31:29.507015+0000","last_fullsized":"2023-10-21T05:31:29.507015+0000","mapping_epoch":14,"log_start":"154'620862","ondisk_log_start":"154'620862","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7726747648,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":620704,"num_read_kb":2454088,"num_write":620778,"num_write_kb":2483112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'16871","reported_seq":24357,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:28.519005+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:31:28.519005+0000","last_peered":"2023-10-21T05:31:28.519005+0000","last_clean":"2023-10-21T05:31:28.519005+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:31:28.519005+0000","last_undegraded":"2023-10-21T05:31:28.519005+0000","last_fullsized":"2023-10-21T05:31:28.519005+0000","mapping_epoch":11,"log_start":"103'6800","ondisk_log_start":"103'6800","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'638200","reported_seq":1263414,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:30.953048+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:31:30.953048+0000","last_peered":"2023-10-21T05:31:30.953048+0000","last_clean":"2023-10-21T05:31:30.953048+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:31:30.953048+0000","last_undegraded":"2023-10-21T05:31:30.953048+0000","last_fullsized":"2023-10-21T05:31:30.953048+0000","mapping_epoch":14,"log_start":"154'628161","ondisk_log_start":"154'628161","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7821684736,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":629541,"num_read_kb":2488576,"num_write":627959,"num_write_kb":2511836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3912,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:26.978227+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:31:26.978227+0000","last_peered":"2023-10-21T05:31:26.978227+0000","last_clean":"2023-10-21T05:31:26.978227+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:31:26.978227+0000","last_undegraded":"2023-10-21T05:31:26.978227+0000","last_fullsized":"2023-10-21T05:31:26.978227+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'667024","reported_seq":1316782,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:29.524724+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:31:29.524724+0000","last_peered":"2023-10-21T05:31:29.524724+0000","last_clean":"2023-10-21T05:31:29.524724+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:31:29.524724+0000","last_undegraded":"2023-10-21T05:31:29.524724+0000","last_fullsized":"2023-10-21T05:31:29.524724+0000","mapping_epoch":14,"log_start":"155'656954","ondisk_log_start":"155'656954","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8170729472,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":654269,"num_read_kb":2587172,"num_write":656310,"num_write_kb":2625240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7768,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:26.979273+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:31:26.979273+0000","last_peered":"2023-10-21T05:31:26.979273+0000","last_clean":"2023-10-21T05:31:26.979273+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:31:26.979273+0000","last_undegraded":"2023-10-21T05:31:26.979273+0000","last_fullsized":"2023-10-21T05:31:26.979273+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'612861","reported_seq":1213790,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:31:30.984924+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:31:30.984924+0000","last_peered":"2023-10-21T05:31:30.984924+0000","last_clean":"2023-10-21T05:31:30.984924+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:31:30.984924+0000","last_undegraded":"2023-10-21T05:31:30.984924+0000","last_fullsized":"2023-10-21T05:31:30.984924+0000","mapping_epoch":14,"log_start":"154'602761","ondisk_log_start":"154'602761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5152,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 819s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7523315731,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":604881,"num_read_kb":2392088,"num_write":603786,"num_write_kb":2415138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'8265","reported_seq":11972,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:30.901853+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:31:30.901853+0000","last_peered":"2023-10-21T05:31:30.901853+0000","last_clean":"2023-10-21T05:31:30.901853+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:31:30.901853+0000","last_undegraded":"2023-10-21T05:31:30.901853+0000","last_fullsized":"2023-10-21T05:31:30.901853+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":339,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:30.556606+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:30:30.556606+0000","last_peered":"2023-10-21T05:30:30.556606+0000","last_clean":"2023-10-21T05:30:30.556606+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:30:30.556606+0000","last_undegraded":"2023-10-21T05:30:30.556606+0000","last_fullsized":"2023-10-21T05:30:30.556606+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31242477587,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2509395,"num_read_kb":9921924,"num_write":2508833,"num_write_kb":10035326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20552650752,"data_stored":40156467238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40280,"ondisk_log_size":40280,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49778,"ondisk_log_size":49778,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739473,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11770896,"kb_used_data":10036616,"kb_used_omap":1562,"kb_used_meta":1732709,"kb_avail":82600944,"statfs":{"total":96636764160,"available":84583366656,"internally_reserved":0,"allocated":10277494784,"data_stored":20078351406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":1774294624},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.86899999999999999,"15min":0.997},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.186},"max":{"1min":3.7749999999999999,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.86699999999999999,"15min":1.0109999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.681,"5min":3.681,"15min":27.251000000000001},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.82199999999999995,"15min":0.86199999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":2.2730000000000001,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.04,"15min":1.006},"min":{"1min":0.25700000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":3.7330000000000001,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.72499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739471,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6529004,"kb_used_data":4928420,"kb_used_omap":1421,"kb_used_meta":1599154,"kb_avail":87842836,"statfs":{"total":96636764160,"available":89951064064,"internally_reserved":0,"allocated":5046702080,"data_stored":9857725442,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1637533951},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:30:34 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":0.90800000000000003,"15min":0.77900000000000003},"min":{"1min":0.223,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.6269999999999998,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.94599999999999995,"15min":0.85599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":4.6609999999999996,"5min":6.968,"15min":6.968},"last":0.73099999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:30:34 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.0960000000000001,"15min":0.93300000000000005},"min":{"1min":0.27000000000000002,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":5.3289999999999997,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.0600000000000001,"15min":0.93600000000000005},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.22},"max":{"1min":5.2880000000000003,"5min":5.9480000000000004,"15min":5.9480000000000004},"last":0.35199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739471,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6973568,"kb_used_data":5109880,"kb_used_omap":1499,"kb_used_meta":1862180,"kb_avail":87398272,"statfs":{"total":96636764160,"available":89495830528,"internally_reserved":0,"allocated":5232517120,"data_stored":10221886681,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":1906872511},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,7,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80300000000000005,"15min":0.77600000000000002},"min":{"1min":0.21199999999999999,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":3.0670000000000002,"5min":4.8090000000000002,"15min":5.923},"last":0.499},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.871,"15min":0.82599999999999996},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":3.1019999999999999,"5min":4.8639999999999999,"15min":5.968},"last":0.53600000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:30:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.0600000000000001,"15min":1.042},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.1210000000000004,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.1160000000000001,"15min":1.0589999999999999},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.0350000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5230952448,"data_stored":10220998656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5045227520,"data_stored":9857089555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10276470784,"data_stored":20078379027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:31:35.794 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:31:35.795 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:31:46.175 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:31:46.177 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:31:46.177 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:31:46.178 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:31:46.178 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:31:46.178 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:31:46.178 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:31:46.179 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:31:46.179 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:31:46.179 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:31:46.168+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:31:52.731 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:52.729+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:31:52.732 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:31:52.729+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:31:55.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:31:56.324 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:31:56.325 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:31:56.347 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2922,"stamp":"2023-10-21T05:31:56.048259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31282910973,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2568221,"num_read_kb":10000334,"num_write":2519716,"num_write_kb":10076639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90153,"ondisk_log_size":90153,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25851532,"kb_used_data":20157428,"kb_used_omap":4483,"kb_used_meta":5689596,"kb_avail":257263988,"statfs":{"total":289910292480,"available":263438323712,"internally_reserved":0,"allocated":20641206272,"data_stored":40242455817,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4591586,"internal_metadata":5826146334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,7,9,15,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22183936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5352,"num_read_kb":21328,"num_write":5418,"num_write_kb":21672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002020"},"pg_stats":[{"pgid":"2.7","version":"156'8332","reported_seq":12116,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:49.422905+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:31:49.422905+0000","last_peered":"2023-10-21T05:31:49.422905+0000","last_clean":"2023-10-21T05:31:49.422905+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:31:49.422905+0000","last_undegraded":"2023-10-21T05:31:49.422905+0000","last_fullsized":"2023-10-21T05:31:49.422905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'4261","reported_seq":8201,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:48.607380+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:31:48.607380+0000","last_peered":"2023-10-21T05:31:48.607380+0000","last_clean":"2023-10-21T05:31:48.607380+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:31:48.607380+0000","last_undegraded":"2023-10-21T05:31:48.607380+0000","last_fullsized":"2023-10-21T05:31:48.607380+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'12440","reported_seq":17968,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:49.460260+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:31:49.460260+0000","last_peered":"2023-10-21T05:31:49.460260+0000","last_clean":"2023-10-21T05:31:49.460260+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:31:49.460260+0000","last_undegraded":"2023-10-21T05:31:49.460260+0000","last_fullsized":"2023-10-21T05:31:49.460260+0000","mapping_epoch":11,"log_start":"70'2400","ondisk_log_start":"70'2400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'8376","reported_seq":13174,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:49.454781+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:31:49.454781+0000","last_peered":"2023-10-21T05:31:49.454781+0000","last_clean":"2023-10-21T05:31:49.454781+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:31:49.454781+0000","last_undegraded":"2023-10-21T05:31:49.454781+0000","last_fullsized":"2023-10-21T05:31:49.454781+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8376,"log_dups_size":0,"ondisk_log_size":8376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'633564","reported_seq":1252536,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:54.716300+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:31:54.716300+0000","last_peered":"2023-10-21T05:31:54.716300+0000","last_clean":"2023-10-21T05:31:54.716300+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:31:54.716300+0000","last_undegraded":"2023-10-21T05:31:54.716300+0000","last_fullsized":"2023-10-21T05:31:54.716300+0000","mapping_epoch":14,"log_start":"155'623562","ondisk_log_start":"155'623562","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7737520128,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":623262,"num_read_kb":2464276,"num_write":623409,"num_write_kb":2493636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'16952","reported_seq":24458,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:54.455464+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:31:54.455464+0000","last_peered":"2023-10-21T05:31:54.455464+0000","last_clean":"2023-10-21T05:31:54.455464+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:31:54.455464+0000","last_undegraded":"2023-10-21T05:31:54.455464+0000","last_fullsized":"2023-10-21T05:31:54.455464+0000","mapping_epoch":11,"log_start":"105'6900","ondisk_log_start":"105'6900","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'640428","reported_seq":1267717,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:51.164420+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:31:51.164420+0000","last_peered":"2023-10-21T05:31:51.164420+0000","last_clean":"2023-10-21T05:31:51.164420+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:31:51.164420+0000","last_undegraded":"2023-10-21T05:31:51.164420+0000","last_fullsized":"2023-10-21T05:31:51.164420+0000","mapping_epoch":14,"log_start":"155'630361","ondisk_log_start":"155'630361","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7830810624,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":631616,"num_read_kb":2496844,"num_write":630187,"num_write_kb":2520748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3923,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:52.576863+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:31:52.576863+0000","last_peered":"2023-10-21T05:31:52.576863+0000","last_clean":"2023-10-21T05:31:52.576863+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:31:52.576863+0000","last_undegraded":"2023-10-21T05:31:52.576863+0000","last_fullsized":"2023-10-21T05:31:52.576863+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'669779","reported_seq":1322243,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:54.693356+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:31:54.693356+0000","last_peered":"2023-10-21T05:31:54.693356+0000","last_clean":"2023-10-21T05:31:54.693356+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:31:54.693356+0000","last_undegraded":"2023-10-21T05:31:54.693356+0000","last_fullsized":"2023-10-21T05:31:54.693356+0000","mapping_epoch":14,"log_start":"155'659754","ondisk_log_start":"155'659754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8182018048,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":656976,"num_read_kb":2597968,"num_write":659065,"num_write_kb":2636260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7788,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:52.586166+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:31:52.586166+0000","last_peered":"2023-10-21T05:31:52.586166+0000","last_clean":"2023-10-21T05:31:52.586166+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:31:52.586166+0000","last_undegraded":"2023-10-21T05:31:52.586166+0000","last_fullsized":"2023-10-21T05:31:52.586166+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'615042","reported_seq":1218010,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:31:51.174727+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:31:51.174727+0000","last_peered":"2023-10-21T05:31:51.174727+0000","last_clean":"2023-10-21T05:31:51.174727+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:31:51.174727+0000","last_undegraded":"2023-10-21T05:31:51.174727+0000","last_fullsized":"2023-10-21T05:31:51.174727+0000","mapping_epoch":14,"log_start":"155'604961","ondisk_log_start":"155'604961","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5235,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 839s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7531966483,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":606908,"num_read_kb":2400176,"num_write":605897,"num_write_kb":2423582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'8289","reported_seq":12004,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:31:49.476931+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:31:49.476931+0000","last_peered":"2023-10-21T05:31:49.476931+0000","last_clean":"2023-10-21T05:31:49.476931+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:31:49.476931+0000","last_undegraded":"2023-10-21T05:31:49.476931+0000","last_fullsized":"2023-10-21T05:31:49.476931+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8289,"log_dups_size":0,"ondisk_log_size":8289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":339,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:30.556606+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:30:30.556606+0000","last_peered":"2023-10-21T05:30:30.556606+0000","last_clean":"2023-10-21T05:30:30.556606+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:30:30.556606+0000","last_undegraded":"2023-10-21T05:30:30.556606+0000","last_fullsized":"2023-10-21T05:30:30.556606+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31282315283,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2518762,"num_read_kb":9959264,"num_write":2518558,"num_write_kb":10074226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20636598272,"data_stored":40240414758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40175,"ondisk_log_size":40175,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49879,"ondisk_log_size":49879,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12070080,"kb_used_data":10079992,"kb_used_omap":1562,"kb_used_meta":1988517,"kb_avail":82301760,"statfs":{"total":96636764160,"available":84277002240,"internally_reserved":0,"allocated":10321911808,"data_stored":20122768430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":2036242016},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.90800000000000003,"15min":0.99099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":3.125,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.747},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.87,"15min":0.96999999999999997},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":1.6220000000000001,"5min":3.681,"15min":27.251000000000001},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Oct 21 05:31:38 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.89200000000000002,"15min":0.88100000000000001},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":3.085,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.085,"15min":1.0009999999999999},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":3.242,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.35999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739475,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6648632,"kb_used_data":4946352,"kb_used_omap":1421,"kb_used_meta":1700850,"kb_avail":87723208,"statfs":{"total":96636764160,"available":89828564992,"internally_reserved":0,"allocated":5065064448,"data_stored":9876087810,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1741670655},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,4,7,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.79700000000000004},"min":{"1min":0.22,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.3170000000000002,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1,"15min":0.86499999999999999},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":2.5019999999999998,"5min":6.968,"15min":6.968},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:31:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":0.93600000000000005},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":2.972,"5min":5.5410000000000004,"15min":5.9950000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.97899999999999998,"15min":0.91600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.22},"max":{"1min":2.1419999999999999,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739476,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7132820,"kb_used_data":5131084,"kb_used_omap":1499,"kb_used_meta":2000228,"kb_avail":87239020,"statfs":{"total":96636764160,"available":89332756480,"internally_reserved":0,"allocated":5254230016,"data_stored":10243599577,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":2048233663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.76000000000000001},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":2.4159999999999999,"5min":4.1260000000000003,"15min":5.923},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.218},"max":{"1min":2.4529999999999998,"5min":4.1600000000000001,"15min":5.968},"last":0.82799999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1060000000000001,"15min":1.016},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.6579999999999999,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.433},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.0720000000000001,"15min":1.026},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.9119999999999999,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5253021696,"data_stored":10243067904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5063004160,"data_stored":9874866195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10320572416,"data_stored":20122480659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:31:56.348 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:31:56.349 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:32:04.403 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:04.395+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:32:04.405 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:04.395+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:32:04.406 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:04.395+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:32:16.351 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:32:16.890 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:32:16.890 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:32:16.915 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2932,"stamp":"2023-10-21T05:32:16.051638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31318447869,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2576583,"num_read_kb":10033642,"num_write":2528385,"num_write_kb":10111315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90189,"ondisk_log_size":90189,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":26274988,"kb_used_data":20226388,"kb_used_omap":4483,"kb_used_meta":6044092,"kb_avail":256840532,"statfs":{"total":289910292480,"available":263004704768,"internally_reserved":0,"allocated":20711821312,"data_stored":40313070857,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4591586,"internal_metadata":6189150238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,12,2,9,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22171648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5218,"num_read_kb":20768,"num_write":5406,"num_write_kb":21624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"156'8354","reported_seq":12146,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:08.853645+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:32:08.853645+0000","last_peered":"2023-10-21T05:32:08.853645+0000","last_clean":"2023-10-21T05:32:08.853645+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:32:08.853645+0000","last_undegraded":"2023-10-21T05:32:08.853645+0000","last_fullsized":"2023-10-21T05:32:08.853645+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8354,"log_dups_size":0,"ondisk_log_size":8354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'4274","reported_seq":8219,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:08.292217+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:32:08.292217+0000","last_peered":"2023-10-21T05:32:08.292217+0000","last_clean":"2023-10-21T05:32:08.292217+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:32:08.292217+0000","last_undegraded":"2023-10-21T05:32:08.292217+0000","last_fullsized":"2023-10-21T05:32:08.292217+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'12474","reported_seq":18015,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:08.862057+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:32:08.862057+0000","last_peered":"2023-10-21T05:32:08.862057+0000","last_clean":"2023-10-21T05:32:08.862057+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:32:08.862057+0000","last_undegraded":"2023-10-21T05:32:08.862057+0000","last_fullsized":"2023-10-21T05:32:08.862057+0000","mapping_epoch":11,"log_start":"70'2400","ondisk_log_start":"70'2400","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'8398","reported_seq":13205,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:08.853529+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:32:08.853529+0000","last_peered":"2023-10-21T05:32:08.853529+0000","last_clean":"2023-10-21T05:32:08.853529+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:32:08.853529+0000","last_undegraded":"2023-10-21T05:32:08.853529+0000","last_fullsized":"2023-10-21T05:32:08.853529+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'635721","reported_seq":1256769,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:14.830763+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:32:14.830763+0000","last_peered":"2023-10-21T05:32:14.830763+0000","last_clean":"2023-10-21T05:32:14.830763+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:32:14.830763+0000","last_undegraded":"2023-10-21T05:32:14.830763+0000","last_fullsized":"2023-10-21T05:32:14.830763+0000","mapping_epoch":14,"log_start":"155'625662","ondisk_log_start":"155'625662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7746363392,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":625338,"num_read_kb":2472528,"num_write":625566,"num_write_kb":2502264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'17008","reported_seq":24531,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:14.724824+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:32:14.724824+0000","last_peered":"2023-10-21T05:32:14.724824+0000","last_clean":"2023-10-21T05:32:14.724824+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:32:14.724824+0000","last_undegraded":"2023-10-21T05:32:14.724824+0000","last_fullsized":"2023-10-21T05:32:14.724824+0000","mapping_epoch":11,"log_start":"105'7000","ondisk_log_start":"105'7000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'642575","reported_seq":1271936,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:11.333239+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:32:11.333239+0000","last_peered":"2023-10-21T05:32:11.333239+0000","last_clean":"2023-10-21T05:32:11.333239+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:32:11.333239+0000","last_undegraded":"2023-10-21T05:32:11.333239+0000","last_fullsized":"2023-10-21T05:32:11.333239+0000","mapping_epoch":14,"log_start":"155'632561","ondisk_log_start":"155'632561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7839612928,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":633689,"num_read_kb":2505100,"num_write":632334,"num_write_kb":2529336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3931,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:13.094497+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:32:13.094497+0000","last_peered":"2023-10-21T05:32:13.094497+0000","last_clean":"2023-10-21T05:32:13.094497+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:32:13.094497+0000","last_undegraded":"2023-10-21T05:32:13.094497+0000","last_fullsized":"2023-10-21T05:32:13.094497+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'672078","reported_seq":1326676,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:14.830236+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:32:14.830236+0000","last_peered":"2023-10-21T05:32:14.830236+0000","last_clean":"2023-10-21T05:32:14.830236+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:32:14.830236+0000","last_undegraded":"2023-10-21T05:32:14.830236+0000","last_fullsized":"2023-10-21T05:32:14.830236+0000","mapping_epoch":14,"log_start":"155'662054","ondisk_log_start":"155'662054","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8191447040,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":659112,"num_read_kb":2606480,"num_write":661364,"num_write_kb":2645456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7805,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:13.064587+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:32:13.064587+0000","last_peered":"2023-10-21T05:32:13.064587+0000","last_clean":"2023-10-21T05:32:13.064587+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:32:13.064587+0000","last_undegraded":"2023-10-21T05:32:13.064587+0000","last_fullsized":"2023-10-21T05:32:13.064587+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'617205","reported_seq":1222258,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:32:11.363548+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:32:11.363548+0000","last_peered":"2023-10-21T05:32:11.363548+0000","last_clean":"2023-10-21T05:32:11.363548+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:32:11.363548+0000","last_undegraded":"2023-10-21T05:32:11.363548+0000","last_fullsized":"2023-10-21T05:32:11.363548+0000","mapping_epoch":14,"log_start":"155'607161","ondisk_log_start":"155'607161","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5357,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 859s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7540428819,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":608985,"num_read_kb":2408464,"num_write":607963,"num_write_kb":2431846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'8312","reported_seq":12036,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-21T05:32:08.858816+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:32:08.858816+0000","last_peered":"2023-10-21T05:32:08.858816+0000","last_clean":"2023-10-21T05:32:08.858816+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:32:08.858816+0000","last_undegraded":"2023-10-21T05:32:08.858816+0000","last_fullsized":"2023-10-21T05:32:08.858816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":339,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-21T05:30:30.556606+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:30:30.556606+0000","last_peered":"2023-10-21T05:30:30.556606+0000","last_clean":"2023-10-21T05:30:30.556606+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:30:30.556606+0000","last_undegraded":"2023-10-21T05:30:30.556606+0000","last_fullsized":"2023-10-21T05:30:30.556606+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31317852179,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2527124,"num_read_kb":9992572,"num_write":2527227,"num_write_kb":10108902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20707840000,"data_stored":40311656486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40141,"ondisk_log_size":40141,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":49949,"ondisk_log_size":49949,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739482,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12284844,"kb_used_data":10114404,"kb_used_omap":1562,"kb_used_meta":2168869,"kb_avail":82086996,"statfs":{"total":96636764160,"available":84057083904,"internally_reserved":0,"allocated":10357149696,"data_stored":20158006318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599904,"internal_metadata":2220922464},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.90800000000000003,"15min":0.99099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":3.125,"5min":8.3439999999999994,"15min":27.184999999999999},"last":1.242},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.87,"15min":0.96999999999999997},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":1.6220000000000001,"5min":3.681,"15min":27.251000000000001},"last":0.246}]},{"osd":1,"last update":"Sat Oct 21 05:31:38 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.89200000000000002,"15min":0.88100000000000001},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":3.085,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.085,"15min":1.0009999999999999},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":3.242,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":1.169}]}]},{"osd":1,"up_from":8,"seq":34359739479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6744984,"kb_used_data":4962960,"kb_used_omap":1421,"kb_used_meta":1780594,"kb_avail":87626856,"statfs":{"total":96636764160,"available":89729900544,"internally_reserved":0,"allocated":5082071040,"data_stored":9893094402,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1823328511},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,1,6,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.79700000000000004},"min":{"1min":0.22,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.3170000000000002,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":2.6230000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1,"15min":0.86499999999999999},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":2.5019999999999998,"5min":6.968,"15min":6.968},"last":0.27600000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:31:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":0.93600000000000005},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":2.972,"5min":5.5410000000000004,"15min":5.9950000000000001},"last":0.63},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.97899999999999998,"15min":0.91600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.22},"max":{"1min":2.1419999999999999,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.59299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739480,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7245160,"kb_used_data":5149024,"kb_used_omap":1499,"kb_used_meta":2094628,"kb_avail":87126680,"statfs":{"total":96636764160,"available":89217720320,"internally_reserved":0,"allocated":5272600576,"data_stored":10261970137,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":2144899263},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.76000000000000001},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":2.4159999999999999,"5min":4.1260000000000003,"15min":5.923},"last":0.31},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.218},"max":{"1min":2.4529999999999998,"5min":4.1600000000000001,"15min":5.968},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1060000000000001,"15min":1.016},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.6579999999999999,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.0720000000000001,"15min":1.026},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.9119999999999999,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5271277568,"data_stored":10261323776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5080276992,"data_stored":9892139027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10356285440,"data_stored":20158193683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:32:16.917 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:32:16.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:32:17.449 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:32:17.472 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:32:17.858 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:17.855+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:32:17.868 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:17.866+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:32:33.650 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.643+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:32:33.652 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.643+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:32:33.652 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.647+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:32:33.652 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.647+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:32:33.653 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.647+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:32:33.653 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.648+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:32:33.658 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.653+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:32:33.658 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.653+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:32:33.658 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.653+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:32:33.659 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:32:33.653+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:32:37.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:32:37.709 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:32:37.706+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:32:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:32:38.026 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:32:38.046 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2943,"stamp":"2023-10-21T05:32:36.055067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31354517245,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2584677,"num_read_kb":10065902,"num_write":2537182,"num_write_kb":10146503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90359,"ondisk_log_size":90359,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25911648,"kb_used_data":20296008,"kb_used_omap":4485,"kb_used_meta":5611130,"kb_avail":257203872,"statfs":{"total":289910292480,"available":263376764928,"internally_reserved":0,"allocated":20783112192,"data_stored":40384349179,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4593266,"internal_metadata":5745797518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,4,9,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18010112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4165,"num_read_kb":16604,"num_write":4394,"num_write_kb":17576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001771"},"pg_stats":[{"pgid":"2.7","version":"157'8378","reported_seq":12179,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:30.046710+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:32:30.046710+0000","last_peered":"2023-10-21T05:32:30.046710+0000","last_clean":"2023-10-21T05:32:30.046710+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:32:30.046710+0000","last_undegraded":"2023-10-21T05:32:30.046710+0000","last_fullsized":"2023-10-21T05:32:30.046710+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8378,"log_dups_size":0,"ondisk_log_size":8378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'4287","reported_seq":8237,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:29.659636+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:32:29.659636+0000","last_peered":"2023-10-21T05:32:29.659636+0000","last_clean":"2023-10-21T05:32:29.659636+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:32:29.659636+0000","last_undegraded":"2023-10-21T05:32:29.659636+0000","last_fullsized":"2023-10-21T05:32:29.659636+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'12510","reported_seq":18064,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:30.056482+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:32:30.056482+0000","last_peered":"2023-10-21T05:32:30.056482+0000","last_clean":"2023-10-21T05:32:30.056482+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:32:30.056482+0000","last_undegraded":"2023-10-21T05:32:30.056482+0000","last_fullsized":"2023-10-21T05:32:30.056482+0000","mapping_epoch":11,"log_start":"70'2500","ondisk_log_start":"70'2500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'8423","reported_seq":13239,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:30.045079+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:32:30.045079+0000","last_peered":"2023-10-21T05:32:30.045079+0000","last_clean":"2023-10-21T05:32:30.045079+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:32:30.045079+0000","last_undegraded":"2023-10-21T05:32:30.045079+0000","last_fullsized":"2023-10-21T05:32:30.045079+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8423,"log_dups_size":0,"ondisk_log_size":8423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'637856","reported_seq":1260839,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:34.975401+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:32:34.975401+0000","last_peered":"2023-10-21T05:32:34.975401+0000","last_clean":"2023-10-21T05:32:34.975401+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:32:34.975401+0000","last_undegraded":"2023-10-21T05:32:34.975401+0000","last_fullsized":"2023-10-21T05:32:34.975401+0000","mapping_epoch":14,"log_start":"155'627762","ondisk_log_start":"155'627762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7755108352,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":627273,"num_read_kb":2480252,"num_write":627701,"num_write_kb":2510804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'17056","reported_seq":24596,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:34.698291+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:32:34.698291+0000","last_peered":"2023-10-21T05:32:34.698291+0000","last_clean":"2023-10-21T05:32:34.698291+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:32:34.698291+0000","last_undegraded":"2023-10-21T05:32:34.698291+0000","last_fullsized":"2023-10-21T05:32:34.698291+0000","mapping_epoch":11,"log_start":"105'7000","ondisk_log_start":"105'7000","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'644825","reported_seq":1276187,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:31.710533+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:32:31.710533+0000","last_peered":"2023-10-21T05:32:31.710533+0000","last_clean":"2023-10-21T05:32:31.710533+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:32:31.710533+0000","last_undegraded":"2023-10-21T05:32:31.710533+0000","last_fullsized":"2023-10-21T05:32:31.710533+0000","mapping_epoch":14,"log_start":"155'634761","ondisk_log_start":"155'634761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7848841216,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":635693,"num_read_kb":2513092,"num_write":634584,"num_write_kb":2538336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3940,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:33.536829+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:32:33.536829+0000","last_peered":"2023-10-21T05:32:33.536829+0000","last_clean":"2023-10-21T05:32:33.536829+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:32:33.536829+0000","last_undegraded":"2023-10-21T05:32:33.536829+0000","last_fullsized":"2023-10-21T05:32:33.536829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'674405","reported_seq":1331186,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:34.939924+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:32:34.939924+0000","last_peered":"2023-10-21T05:32:34.939924+0000","last_clean":"2023-10-21T05:32:34.939924+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:32:34.939924+0000","last_undegraded":"2023-10-21T05:32:34.939924+0000","last_fullsized":"2023-10-21T05:32:34.939924+0000","mapping_epoch":14,"log_start":"155'664354","ondisk_log_start":"155'664354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8201003008,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":661296,"num_read_kb":2615192,"num_write":663691,"num_write_kb":2654764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7822,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:33.541991+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:32:33.541991+0000","last_peered":"2023-10-21T05:32:33.541991+0000","last_clean":"2023-10-21T05:32:33.541991+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:32:33.541991+0000","last_undegraded":"2023-10-21T05:32:33.541991+0000","last_fullsized":"2023-10-21T05:32:33.541991+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'619393","reported_seq":1226429,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:32:31.716395+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:32:31.716395+0000","last_peered":"2023-10-21T05:32:31.716395+0000","last_clean":"2023-10-21T05:32:31.716395+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:32:31.716395+0000","last_undegraded":"2023-10-21T05:32:31.716395+0000","last_fullsized":"2023-10-21T05:32:31.716395+0000","mapping_epoch":14,"log_start":"155'609361","ondisk_log_start":"155'609361","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5474,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 880s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7548968979,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":610956,"num_read_kb":2416296,"num_write":610048,"num_write_kb":2440186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'8336","reported_seq":12069,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:30.053276+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:32:30.053276+0000","last_peered":"2023-10-21T05:32:30.053276+0000","last_clean":"2023-10-21T05:32:30.053276+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:32:30.053276+0000","last_undegraded":"2023-10-21T05:32:30.053276+0000","last_fullsized":"2023-10-21T05:32:30.053276+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":341,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:29.664901+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:32:29.664901+0000","last_peered":"2023-10-21T05:32:29.664901+0000","last_clean":"2023-10-21T05:32:29.664901+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:32:29.664901+0000","last_undegraded":"2023-10-21T05:32:29.664901+0000","last_fullsized":"2023-10-21T05:32:29.664901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31353921555,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2535218,"num_read_kb":10024832,"num_write":2536024,"num_write_kb":10144090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20779311104,"data_stored":40383127590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50019,"ondisk_log_size":50019,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739486,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11868596,"kb_used_data":10149164,"kb_used_omap":1564,"kb_used_meta":1717859,"kb_avail":82503244,"statfs":{"total":96636764160,"available":84483321856,"internally_reserved":0,"allocated":10392743936,"data_stored":20193596372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1601584,"internal_metadata":1759088592},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.90800000000000003,"15min":0.99099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":3.125,"5min":8.3439999999999994,"15min":27.184999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.87,"15min":0.96999999999999997},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":1.6220000000000001,"5min":3.681,"15min":27.251000000000001},"last":0.38}]},{"osd":1,"last update":"Sat Oct 21 05:31:38 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.89200000000000002,"15min":0.88100000000000001},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":3.085,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.085,"15min":1.0009999999999999},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":3.242,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359739483,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6852376,"kb_used_data":4980112,"kb_used_omap":1421,"kb_used_meta":1870834,"kb_avail":87519464,"statfs":{"total":96636764160,"available":89619931136,"internally_reserved":0,"allocated":5099634688,"data_stored":9910653864,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1915734271},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.79700000000000004},"min":{"1min":0.22,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.3170000000000002,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1,"15min":0.86499999999999999},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":2.5019999999999998,"5min":6.968,"15min":6.968},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:31:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":0.93600000000000005},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":2.972,"5min":5.5410000000000004,"15min":5.9950000000000001},"last":1.004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.97899999999999998,"15min":0.91600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.22},"max":{"1min":2.1419999999999999,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.95199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739484,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7190676,"kb_used_data":5166732,"kb_used_omap":1499,"kb_used_meta":2022436,"kb_avail":87181164,"statfs":{"total":96636764160,"available":89273511936,"internally_reserved":0,"allocated":5290733568,"data_stored":10280098943,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":2070974655},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.76000000000000001},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":2.4159999999999999,"5min":4.1260000000000003,"15min":5.923},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.218},"max":{"1min":2.4529999999999998,"5min":4.1600000000000001,"15min":5.968},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1060000000000001,"15min":1.016},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.6579999999999999,"5min":7.1210000000000004,"15min":14.053000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.0720000000000001,"15min":1.026},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.9119999999999999,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5289549824,"data_stored":10279596032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5098008576,"data_stored":9909870611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10391752704,"data_stored":20193660947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:32:38.048 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:32:38.049 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:32:53.005 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:32:53.001+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:32:53.006 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:32:53.001+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:32:53.006 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:32:53.001+0000 7f1e40431640 -1 reset not still connected to 0x56047b722c30 2023-10-21T05:32:53.006 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:32:53.001+0000 7f1e40431640 -1 reset not still connected to 0x56047c28ec30 2023-10-21T05:32:58.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:32:58.603 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:32:58.604 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:32:58.624 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2954,"stamp":"2023-10-21T05:32:58.058850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31394985725,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2594464,"num_read_kb":10104902,"num_write":2547059,"num_write_kb":10186011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90381,"ondisk_log_size":90381,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":26363932,"kb_used_data":20370628,"kb_used_omap":4485,"kb_used_meta":5988794,"kb_avail":256751588,"statfs":{"total":289910292480,"available":262913626112,"internally_reserved":0,"allocated":20859523072,"data_stored":40460760059,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4593266,"internal_metadata":6132525454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,7,10,15,37],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22335488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5425,"num_read_kb":21616,"num_write":5450,"num_write_kb":21800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002040"},"pg_stats":[{"pgid":"2.7","version":"157'8412","reported_seq":12223,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:55.642684+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:32:55.642684+0000","last_peered":"2023-10-21T05:32:55.642684+0000","last_clean":"2023-10-21T05:32:55.642684+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:32:55.642684+0000","last_undegraded":"2023-10-21T05:32:55.642684+0000","last_fullsized":"2023-10-21T05:32:55.642684+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'4304","reported_seq":8259,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:54.824247+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:32:54.824247+0000","last_peered":"2023-10-21T05:32:54.824247+0000","last_clean":"2023-10-21T05:32:54.824247+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:32:54.824247+0000","last_undegraded":"2023-10-21T05:32:54.824247+0000","last_fullsized":"2023-10-21T05:32:54.824247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'12561","reported_seq":18130,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:55.655553+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:32:55.655553+0000","last_peered":"2023-10-21T05:32:55.655553+0000","last_clean":"2023-10-21T05:32:55.655553+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:32:55.655553+0000","last_undegraded":"2023-10-21T05:32:55.655553+0000","last_fullsized":"2023-10-21T05:32:55.655553+0000","mapping_epoch":11,"log_start":"70'2500","ondisk_log_start":"70'2500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'8457","reported_seq":13283,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:55.648107+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:32:55.648107+0000","last_peered":"2023-10-21T05:32:55.648107+0000","last_clean":"2023-10-21T05:32:55.648107+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:32:55.648107+0000","last_undegraded":"2023-10-21T05:32:55.648107+0000","last_fullsized":"2023-10-21T05:32:55.648107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'640109","reported_seq":1265306,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:55.281502+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:32:55.281502+0000","last_peered":"2023-10-21T05:32:55.281502+0000","last_clean":"2023-10-21T05:32:55.281502+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:32:55.281502+0000","last_undegraded":"2023-10-21T05:32:55.281502+0000","last_fullsized":"2023-10-21T05:32:55.281502+0000","mapping_epoch":14,"log_start":"156'630062","ondisk_log_start":"156'630062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7764348928,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":629486,"num_read_kb":2489044,"num_write":629954,"num_write_kb":2519816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'17121","reported_seq":24677,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:54.244517+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:32:54.244517+0000","last_peered":"2023-10-21T05:32:54.244517+0000","last_clean":"2023-10-21T05:32:54.244517+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:32:54.244517+0000","last_undegraded":"2023-10-21T05:32:54.244517+0000","last_fullsized":"2023-10-21T05:32:54.244517+0000","mapping_epoch":11,"log_start":"107'7100","ondisk_log_start":"107'7100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'647578","reported_seq":1281586,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:56.894368+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:32:56.894368+0000","last_peered":"2023-10-21T05:32:56.894368+0000","last_clean":"2023-10-21T05:32:56.894368+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:32:56.894368+0000","last_undegraded":"2023-10-21T05:32:56.894368+0000","last_fullsized":"2023-10-21T05:32:56.894368+0000","mapping_epoch":14,"log_start":"156'637561","ondisk_log_start":"156'637561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7860121600,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":638336,"num_read_kb":2523640,"num_write":637337,"num_write_kb":2549348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3948,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:54.020575+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:32:54.020575+0000","last_peered":"2023-10-21T05:32:54.020575+0000","last_clean":"2023-10-21T05:32:54.020575+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:32:54.020575+0000","last_undegraded":"2023-10-21T05:32:54.020575+0000","last_fullsized":"2023-10-21T05:32:54.020575+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'676711","reported_seq":1335856,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:55.274286+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:32:55.274286+0000","last_peered":"2023-10-21T05:32:55.274286+0000","last_clean":"2023-10-21T05:32:55.274286+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:32:55.274286+0000","last_undegraded":"2023-10-21T05:32:55.274286+0000","last_fullsized":"2023-10-21T05:32:55.274286+0000","mapping_epoch":14,"log_start":"156'666654","ondisk_log_start":"156'666654","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8210448384,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":663653,"num_read_kb":2624592,"num_write":665997,"num_write_kb":2663988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7838,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:54.018320+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:32:54.018320+0000","last_peered":"2023-10-21T05:32:54.018320+0000","last_clean":"2023-10-21T05:32:54.018320+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:32:54.018320+0000","last_undegraded":"2023-10-21T05:32:54.018320+0000","last_fullsized":"2023-10-21T05:32:54.018320+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'622068","reported_seq":1231684,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:32:56.904148+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:32:56.904148+0000","last_peered":"2023-10-21T05:32:56.904148+0000","last_clean":"2023-10-21T05:32:56.904148+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:32:56.904148+0000","last_undegraded":"2023-10-21T05:32:56.904148+0000","last_fullsized":"2023-10-21T05:32:56.904148+0000","mapping_epoch":14,"log_start":"156'612061","ondisk_log_start":"156'612061","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5603,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 905s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7559471123,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":613530,"num_read_kb":2426556,"num_write":612613,"num_write_kb":2450446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'8370","reported_seq":12113,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:55.672532+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:32:55.672532+0000","last_peered":"2023-10-21T05:32:55.672532+0000","last_clean":"2023-10-21T05:32:55.672532+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:32:55.672532+0000","last_undegraded":"2023-10-21T05:32:55.672532+0000","last_fullsized":"2023-10-21T05:32:55.672532+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":341,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:29.664901+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:32:29.664901+0000","last_peered":"2023-10-21T05:32:29.664901+0000","last_clean":"2023-10-21T05:32:29.664901+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:32:29.664901+0000","last_undegraded":"2023-10-21T05:32:29.664901+0000","last_fullsized":"2023-10-21T05:32:29.664901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31394390035,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2545005,"num_read_kb":10063832,"num_write":2545901,"num_write_kb":10183598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20855603200,"data_stored":40459419686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40128,"ondisk_log_size":40128,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50154,"ondisk_log_size":50154,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739490,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12094196,"kb_used_data":10183852,"kb_used_omap":1564,"kb_used_meta":1908771,"kb_avail":82277644,"statfs":{"total":96636764160,"available":84252307456,"internally_reserved":0,"allocated":10428264448,"data_stored":20229116884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1601584,"internal_metadata":1954582480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:32:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92000000000000004,"15min":0.90100000000000002},"min":{"1min":0.248,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":2.1909999999999998,"5min":8.3439999999999994,"15min":8.3439999999999994},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.087,"5min":0.89700000000000002,"15min":0.89400000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.4119999999999999,"5min":3.681,"15min":6.306},"last":0.77200000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:32:39 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.90600000000000003,"15min":0.89700000000000002},"min":{"1min":0.251,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.7970000000000002,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0720000000000001,"15min":0.99199999999999999},"min":{"1min":0.26700000000000002,"5min":0.23400000000000001,"15min":0.221},"max":{"1min":3.6629999999999998,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.84399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739488,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6962464,"kb_used_data":5001112,"kb_used_omap":1421,"kb_used_meta":1959922,"kb_avail":87409376,"statfs":{"total":96636764160,"available":89507201024,"internally_reserved":0,"allocated":5121138688,"data_stored":9932157864,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":2006960383},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,5,8,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:32:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91500000000000004,"15min":0.79400000000000004},"min":{"1min":0.19,"5min":0.19,"15min":0.17199999999999999},"max":{"1min":3.5800000000000001,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.879,"15min":0.85899999999999999},"min":{"1min":0.22600000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":3.6080000000000001,"5min":5.5019999999999998,"15min":6.968},"last":0.46200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:32:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.006,"15min":0.92100000000000004},"min":{"1min":0.25800000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":3.6829999999999998,"5min":5.5410000000000004,"15min":5.9950000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.90200000000000002,"15min":0.90800000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":2.6389999999999998,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359739488,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7307272,"kb_used_data":5185664,"kb_used_omap":1499,"kb_used_meta":2120100,"kb_avail":87064568,"statfs":{"total":96636764160,"available":89154117632,"internally_reserved":0,"allocated":5310119936,"data_stored":10299485311,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1535809,"internal_metadata":2170982591},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,21],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:32:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":0.75600000000000001},"min":{"1min":0.26400000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":2.2730000000000001,"5min":4.1260000000000003,"15min":5.923},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.77200000000000002},"min":{"1min":0.29999999999999999,"5min":0.27800000000000002,"15min":0.223},"max":{"1min":2.306,"5min":4.1600000000000001,"15min":5.968},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:32:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.0169999999999999,"15min":1.0109999999999999},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.3720000000000001,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.255},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.1140000000000001,"15min":1.044},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.4830000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":1.0189999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5308215296,"data_stored":10298261504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5119815680,"data_stored":9931677715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10427572224,"data_stored":20229480467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:32:58.625 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:32:58.626 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:33:06.543 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:33:06.544 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:33:06.544 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:33:06.544 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:33:06.545 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:33:06.545 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:33:06.545 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:33:06.545 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:33:06.546 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:33:06.546 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:06.540+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:33:18.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:33:19.183 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:33:19.184 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:33:19.204 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2964,"stamp":"2023-10-21T05:33:18.062175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31431247613,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2602851,"num_read_kb":10138266,"num_write":2555898,"num_write_kb":10221367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90589,"ondisk_log_size":90589,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25481296,"kb_used_data":20440312,"kb_used_omap":4487,"kb_used_meta":5036472,"kb_avail":257634224,"statfs":{"total":289910292480,"available":263817445376,"internally_reserved":0,"allocated":20930879488,"data_stored":40532116475,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4594946,"internal_metadata":5157348094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,6,11,7,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22528000,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5185,"num_read_kb":20624,"num_write":5494,"num_write_kb":21976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002040"},"pg_stats":[{"pgid":"2.7","version":"157'8438","reported_seq":12257,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:16.556881+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:33:16.556881+0000","last_peered":"2023-10-21T05:33:16.556881+0000","last_clean":"2023-10-21T05:33:16.556881+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:33:16.556881+0000","last_undegraded":"2023-10-21T05:33:16.556881+0000","last_fullsized":"2023-10-21T05:33:16.556881+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8438,"log_dups_size":0,"ondisk_log_size":8438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'4317","reported_seq":8276,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:15.554807+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:33:15.554807+0000","last_peered":"2023-10-21T05:33:15.554807+0000","last_clean":"2023-10-21T05:33:15.554807+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:33:15.554807+0000","last_undegraded":"2023-10-21T05:33:15.554807+0000","last_fullsized":"2023-10-21T05:33:15.554807+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'12600","reported_seq":18181,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:16.568466+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:33:16.568466+0000","last_peered":"2023-10-21T05:33:16.568466+0000","last_clean":"2023-10-21T05:33:16.568466+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:33:16.568466+0000","last_undegraded":"2023-10-21T05:33:16.568466+0000","last_fullsized":"2023-10-21T05:33:16.568466+0000","mapping_epoch":11,"log_start":"70'2500","ondisk_log_start":"70'2500","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'8483","reported_seq":13317,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:16.547557+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:33:16.547557+0000","last_peered":"2023-10-21T05:33:16.547557+0000","last_clean":"2023-10-21T05:33:16.547557+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:33:16.547557+0000","last_undegraded":"2023-10-21T05:33:16.547557+0000","last_fullsized":"2023-10-21T05:33:16.547557+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8483,"log_dups_size":0,"ondisk_log_size":8483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'642287","reported_seq":1269580,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:15.364974+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:33:15.364974+0000","last_peered":"2023-10-21T05:33:15.364974+0000","last_clean":"2023-10-21T05:33:15.364974+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:33:15.364974+0000","last_undegraded":"2023-10-21T05:33:15.364974+0000","last_fullsized":"2023-10-21T05:33:15.364974+0000","mapping_epoch":14,"log_start":"156'632262","ondisk_log_start":"156'632262","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7773282304,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":631581,"num_read_kb":2497388,"num_write":632132,"num_write_kb":2528528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'17185","reported_seq":24757,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:15.240620+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:33:15.240620+0000","last_peered":"2023-10-21T05:33:15.240620+0000","last_clean":"2023-10-21T05:33:15.240620+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:33:15.240620+0000","last_undegraded":"2023-10-21T05:33:15.240620+0000","last_fullsized":"2023-10-21T05:33:15.240620+0000","mapping_epoch":11,"log_start":"107'7100","ondisk_log_start":"107'7100","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'649765","reported_seq":1285910,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:17.058745+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:33:17.058745+0000","last_peered":"2023-10-21T05:33:17.058745+0000","last_clean":"2023-10-21T05:33:17.058745+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:33:17.058745+0000","last_undegraded":"2023-10-21T05:33:17.058745+0000","last_fullsized":"2023-10-21T05:33:17.058745+0000","mapping_epoch":14,"log_start":"156'639761","ondisk_log_start":"156'639761","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7869091840,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":640470,"num_read_kb":2532120,"num_write":639524,"num_write_kb":2558096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3956,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:14.506342+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:33:14.506342+0000","last_peered":"2023-10-21T05:33:14.506342+0000","last_clean":"2023-10-21T05:33:14.506342+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:33:14.506342+0000","last_undegraded":"2023-10-21T05:33:14.506342+0000","last_fullsized":"2023-10-21T05:33:14.506342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'679069","reported_seq":1340473,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:15.349453+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:33:15.349453+0000","last_peered":"2023-10-21T05:33:15.349453+0000","last_clean":"2023-10-21T05:33:15.349453+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:33:15.349453+0000","last_undegraded":"2023-10-21T05:33:15.349453+0000","last_fullsized":"2023-10-21T05:33:15.349453+0000","mapping_epoch":14,"log_start":"156'669054","ondisk_log_start":"156'669054","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8220131328,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":665913,"num_read_kb":2633564,"num_write":668355,"num_write_kb":2673420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7854,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:14.497780+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:33:14.497780+0000","last_peered":"2023-10-21T05:33:14.497780+0000","last_clean":"2023-10-21T05:33:14.497780+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:33:14.497780+0000","last_undegraded":"2023-10-21T05:33:14.497780+0000","last_fullsized":"2023-10-21T05:33:14.497780+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'624259","reported_seq":1235784,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:33:17.076926+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:33:17.076926+0000","last_peered":"2023-10-21T05:33:17.076926+0000","last_clean":"2023-10-21T05:33:17.076926+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:33:17.076926+0000","last_undegraded":"2023-10-21T05:33:17.076926+0000","last_fullsized":"2023-10-21T05:33:17.076926+0000","mapping_epoch":14,"log_start":"156'614161","ondisk_log_start":"156'614161","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5690,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 925s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7568146451,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":615428,"num_read_kb":2434124,"num_write":614729,"num_write_kb":2458910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'8396","reported_seq":12147,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:33:16.565883+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:33:16.565883+0000","last_peered":"2023-10-21T05:33:16.565883+0000","last_clean":"2023-10-21T05:33:16.565883+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:33:16.565883+0000","last_undegraded":"2023-10-21T05:33:16.565883+0000","last_fullsized":"2023-10-21T05:33:16.565883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8396,"log_dups_size":0,"ondisk_log_size":8396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":341,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:29.664901+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:32:29.664901+0000","last_peered":"2023-10-21T05:32:29.664901+0000","last_clean":"2023-10-21T05:32:29.664901+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:32:29.664901+0000","last_undegraded":"2023-10-21T05:32:29.664901+0000","last_fullsized":"2023-10-21T05:32:29.664901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31430651923,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2553392,"num_read_kb":10097196,"num_write":2554740,"num_write_kb":10218954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20927758336,"data_stored":40531574822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40142,"ondisk_log_size":40142,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50348,"ondisk_log_size":50348,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739494,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12291704,"kb_used_data":10218928,"kb_used_omap":1564,"kb_used_meta":2071203,"kb_avail":82080136,"statfs":{"total":96636764160,"available":84050059264,"internally_reserved":0,"allocated":10464182272,"data_stored":20265034708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1601584,"internal_metadata":2120912848},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:32:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92000000000000004,"15min":0.90100000000000002},"min":{"1min":0.248,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":2.1909999999999998,"5min":8.3439999999999994,"15min":8.3439999999999994},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.087,"5min":0.89700000000000002,"15min":0.89400000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.4119999999999999,"5min":3.681,"15min":6.306},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:32:39 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.90600000000000003,"15min":0.89700000000000002},"min":{"1min":0.251,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.7970000000000002,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0720000000000001,"15min":0.99199999999999999},"min":{"1min":0.26700000000000002,"5min":0.23400000000000001,"15min":0.221},"max":{"1min":3.6629999999999998,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.60599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739492,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7069944,"kb_used_data":5018352,"kb_used_omap":1421,"kb_used_meta":2050162,"kb_avail":87301896,"statfs":{"total":96636764160,"available":89397141504,"internally_reserved":0,"allocated":5138792448,"data_stored":9949811624,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":2099366143},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,3,12,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:32:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91500000000000004,"15min":0.79400000000000004},"min":{"1min":0.19,"5min":0.19,"15min":0.17199999999999999},"max":{"1min":3.5800000000000001,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.879,"15min":0.85899999999999999},"min":{"1min":0.22600000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":3.6080000000000001,"5min":5.5019999999999998,"15min":6.968},"last":2.202}]},{"osd":2,"last update":"Sat Oct 21 05:32:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.006,"15min":0.92100000000000004},"min":{"1min":0.25800000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":3.6829999999999998,"5min":5.5410000000000004,"15min":5.9950000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.90200000000000002,"15min":0.90800000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":2.6389999999999998,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.77200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739492,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6119648,"kb_used_data":5203032,"kb_used_omap":1501,"kb_used_meta":915106,"kb_avail":88252192,"statfs":{"total":96636764160,"available":90370244608,"internally_reserved":0,"allocated":5327904768,"data_stored":10317270143,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1537489,"internal_metadata":937069103},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:32:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":0.75600000000000001},"min":{"1min":0.26400000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":2.2730000000000001,"5min":4.1260000000000003,"15min":5.923},"last":0.433},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.77200000000000002},"min":{"1min":0.29999999999999999,"5min":0.27800000000000002,"15min":0.223},"max":{"1min":2.306,"5min":4.1600000000000001,"15min":5.968},"last":0.46800000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:32:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.0169999999999999,"15min":1.0109999999999999},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.3720000000000001,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.1140000000000001,"15min":1.044},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.4830000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":2.1000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5326794752,"data_stored":10316840960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5137420288,"data_stored":9949282323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10463543296,"data_stored":20265451539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:33:19.206 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:33:19.206 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:33:25.265 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:33:25.262+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:33:39.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:33:39.732 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:33:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:33:39.752 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2975,"stamp":"2023-10-21T05:33:38.065667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31467370237,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2611779,"num_read_kb":10173810,"num_write":2564706,"num_write_kb":10256599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90506,"ondisk_log_size":90506,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25053348,"kb_used_data":20510468,"kb_used_omap":4521,"kb_used_meta":4538326,"kb_avail":258062172,"statfs":{"total":289910292480,"available":264255664128,"internally_reserved":0,"allocated":21002719232,"data_stored":40603947761,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4630080,"internal_metadata":4647246272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,7,10,11,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18178048,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4544,"num_read_kb":18084,"num_write":4430,"num_write_kb":17720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001715"},"pg_stats":[{"pgid":"2.7","version":"157'8456","reported_seq":12285,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.977736+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:33:34.977736+0000","last_peered":"2023-10-21T05:33:34.977736+0000","last_clean":"2023-10-21T05:33:34.977736+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:33:34.977736+0000","last_undegraded":"2023-10-21T05:33:34.977736+0000","last_fullsized":"2023-10-21T05:33:34.977736+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'4326","reported_seq":8290,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.673426+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:33:34.673426+0000","last_peered":"2023-10-21T05:33:34.673426+0000","last_clean":"2023-10-21T05:33:34.673426+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:33:34.673426+0000","last_undegraded":"2023-10-21T05:33:34.673426+0000","last_fullsized":"2023-10-21T05:33:34.673426+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'12627","reported_seq":18221,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.977176+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:33:34.977176+0000","last_peered":"2023-10-21T05:33:34.977176+0000","last_clean":"2023-10-21T05:33:34.977176+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:33:34.977176+0000","last_undegraded":"2023-10-21T05:33:34.977176+0000","last_fullsized":"2023-10-21T05:33:34.977176+0000","mapping_epoch":11,"log_start":"72'2600","ondisk_log_start":"72'2600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'8501","reported_seq":13344,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.976946+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:33:34.976946+0000","last_peered":"2023-10-21T05:33:34.976946+0000","last_clean":"2023-10-21T05:33:34.976946+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:33:34.976946+0000","last_undegraded":"2023-10-21T05:33:34.976946+0000","last_fullsized":"2023-10-21T05:33:34.976946+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8501,"log_dups_size":0,"ondisk_log_size":8501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'644571","reported_seq":1274228,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:35.476729+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:33:35.476729+0000","last_peered":"2023-10-21T05:33:35.476729+0000","last_clean":"2023-10-21T05:33:35.476729+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:33:35.476729+0000","last_undegraded":"2023-10-21T05:33:35.476729+0000","last_fullsized":"2023-10-21T05:33:35.476729+0000","mapping_epoch":14,"log_start":"156'634562","ondisk_log_start":"156'634562","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7782658048,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":633943,"num_read_kb":2506784,"num_write":634416,"num_write_kb":2537664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'17217","reported_seq":24806,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:35.433539+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:33:35.433539+0000","last_peered":"2023-10-21T05:33:35.433539+0000","last_clean":"2023-10-21T05:33:35.433539+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:33:35.433539+0000","last_undegraded":"2023-10-21T05:33:35.433539+0000","last_fullsized":"2023-10-21T05:33:35.433539+0000","mapping_epoch":11,"log_start":"107'7200","ondisk_log_start":"107'7200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'651898","reported_seq":1290162,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:37.179959+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:33:37.179959+0000","last_peered":"2023-10-21T05:33:37.179959+0000","last_clean":"2023-10-21T05:33:37.179959+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:33:37.179959+0000","last_undegraded":"2023-10-21T05:33:37.179959+0000","last_fullsized":"2023-10-21T05:33:37.179959+0000","mapping_epoch":14,"log_start":"156'641861","ondisk_log_start":"156'641861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7877844992,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":642589,"num_read_kb":2540564,"num_write":641657,"num_write_kb":2566628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3965,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.976798+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:33:34.976798+0000","last_peered":"2023-10-21T05:33:34.976798+0000","last_clean":"2023-10-21T05:33:34.976798+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:33:34.976798+0000","last_undegraded":"2023-10-21T05:33:34.976798+0000","last_fullsized":"2023-10-21T05:33:34.976798+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'681382","reported_seq":1345077,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:35.483702+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:33:35.483702+0000","last_peered":"2023-10-21T05:33:35.483702+0000","last_clean":"2023-10-21T05:33:35.483702+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:33:35.483702+0000","last_undegraded":"2023-10-21T05:33:35.483702+0000","last_fullsized":"2023-10-21T05:33:35.483702+0000","mapping_epoch":14,"log_start":"156'671354","ondisk_log_start":"156'671354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8229613568,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":668203,"num_read_kb":2642692,"num_write":670668,"num_write_kb":2682672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7871,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.977164+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:33:34.977164+0000","last_peered":"2023-10-21T05:33:34.977164+0000","last_clean":"2023-10-21T05:33:34.977164+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:33:34.977164+0000","last_undegraded":"2023-10-21T05:33:34.977164+0000","last_fullsized":"2023-10-21T05:33:34.977164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'626424","reported_seq":1240123,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:33:37.189778+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:33:37.189778+0000","last_peered":"2023-10-21T05:33:37.189778+0000","last_clean":"2023-10-21T05:33:37.189778+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:33:37.189778+0000","last_undegraded":"2023-10-21T05:33:37.189778+0000","last_fullsized":"2023-10-21T05:33:37.189778+0000","mapping_epoch":14,"log_start":"156'616361","ondisk_log_start":"156'616361","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5791,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 945s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7576657939,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":617585,"num_read_kb":2442700,"num_write":616807,"num_write_kb":2467222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'8414","reported_seq":12174,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.976930+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:33:34.976930+0000","last_peered":"2023-10-21T05:33:34.976930+0000","last_clean":"2023-10-21T05:33:34.976930+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:33:34.976930+0000","last_undegraded":"2023-10-21T05:33:34.976930+0000","last_fullsized":"2023-10-21T05:33:34.976930+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":342,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:31.711284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:32:31.711284+0000","last_peered":"2023-10-21T05:32:31.711284+0000","last_clean":"2023-10-21T05:32:31.711284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:32:31.711284+0000","last_undegraded":"2023-10-21T05:32:31.711284+0000","last_fullsized":"2023-10-21T05:32:31.711284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31466774547,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2562320,"num_read_kb":10132740,"num_write":2563548,"num_write_kb":10254186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21000097792,"data_stored":40603914278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40137,"ondisk_log_size":40137,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50270,"ondisk_log_size":50270,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739498,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12362924,"kb_used_data":10253924,"kb_used_omap":1598,"kb_used_meta":2107393,"kb_avail":82008916,"statfs":{"total":96636764160,"available":83977129984,"internally_reserved":0,"allocated":10500018176,"data_stored":20300866383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1636718,"internal_metadata":2157971090},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,5,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:32:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92000000000000004,"15min":0.90100000000000002},"min":{"1min":0.248,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":2.1909999999999998,"5min":8.3439999999999994,"15min":8.3439999999999994},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.087,"5min":0.89700000000000002,"15min":0.89400000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.4119999999999999,"5min":3.681,"15min":6.306},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:32:39 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.90600000000000003,"15min":0.89700000000000002},"min":{"1min":0.251,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.7970000000000002,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0720000000000001,"15min":0.99199999999999999},"min":{"1min":0.26700000000000002,"5min":0.23400000000000001,"15min":0.221},"max":{"1min":3.6629999999999998,"5min":8.3100000000000005,"15min":8.3100000000000005},"last":0.46100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739496,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6545420,"kb_used_data":5035200,"kb_used_omap":1421,"kb_used_meta":1508786,"kb_avail":87826420,"statfs":{"total":96636764160,"available":89934254080,"internally_reserved":0,"allocated":5156044800,"data_stored":9967059747,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1544997119},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,2,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:32:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91500000000000004,"15min":0.79400000000000004},"min":{"1min":0.19,"5min":0.19,"15min":0.17199999999999999},"max":{"1min":3.5800000000000001,"5min":5.5810000000000004,"15min":5.5810000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.879,"15min":0.85899999999999999},"min":{"1min":0.22600000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":3.6080000000000001,"5min":5.5019999999999998,"15min":6.968},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:32:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.006,"15min":0.92100000000000004},"min":{"1min":0.25800000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":3.6829999999999998,"5min":5.5410000000000004,"15min":5.9950000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.90200000000000002,"15min":0.90800000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":2.6389999999999998,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6145004,"kb_used_data":5221344,"kb_used_omap":1501,"kb_used_meta":922146,"kb_avail":88226836,"statfs":{"total":96636764160,"available":90344280064,"internally_reserved":0,"allocated":5346656256,"data_stored":10336021631,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1537489,"internal_metadata":944278063},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:32:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":0.75600000000000001},"min":{"1min":0.26400000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":2.2730000000000001,"5min":4.1260000000000003,"15min":5.923},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.77200000000000002},"min":{"1min":0.29999999999999999,"5min":0.27800000000000002,"15min":0.223},"max":{"1min":2.306,"5min":4.1600000000000001,"15min":5.968},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:32:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.0169999999999999,"15min":1.0109999999999999},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.3720000000000001,"5min":7.1210000000000004,"15min":14.053000000000001},"last":3.5419999999999998},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.1140000000000001,"15min":1.044},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.4830000000000001,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":1.2789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5345628160,"data_stored":10335674368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5154680832,"data_stored":9966542867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10499788800,"data_stored":20301697043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:33:39.754 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:33:39.755 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:33:39.951 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:33:39.948+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:33:39.951 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:33:39.948+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:33:42.275 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:33:42.275 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:33:42.275 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:33:42.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:33:42.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:33:42.276 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:33:42.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:33:42.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:33:42.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:33:42.277 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:33:42.271+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:33:43.844 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:33:43.837+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:33:44.598 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:33:44.593+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:33:59.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:34:00.275 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:34:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:34:00.300 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2986,"stamp":"2023-10-21T05:34:00.069489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31503783677,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2620563,"num_read_kb":10208782,"num_write":2573588,"num_write_kb":10292127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90741,"ondisk_log_size":90741,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24258608,"kb_used_data":20582040,"kb_used_omap":4521,"kb_used_meta":3672022,"kb_avail":258856912,"statfs":{"total":289910292480,"available":265069477888,"internally_reserved":0,"allocated":21076008960,"data_stored":40677233260,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4630080,"internal_metadata":3760150976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,5,9,17,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17936384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4256,"num_read_kb":16952,"num_write":4377,"num_write_kb":17508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002069"},"pg_stats":[{"pgid":"2.7","version":"158'8488","reported_seq":12325,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:56.696119+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:33:56.696119+0000","last_peered":"2023-10-21T05:33:56.696119+0000","last_clean":"2023-10-21T05:33:56.696119+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:33:56.696119+0000","last_undegraded":"2023-10-21T05:33:56.696119+0000","last_fullsized":"2023-10-21T05:33:56.696119+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'4342","reported_seq":8310,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:56.585436+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:33:56.585436+0000","last_peered":"2023-10-21T05:33:56.585436+0000","last_clean":"2023-10-21T05:33:56.585436+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:33:56.585436+0000","last_undegraded":"2023-10-21T05:33:56.585436+0000","last_fullsized":"2023-10-21T05:33:56.585436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'12675","reported_seq":18281,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:56.709046+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:33:56.709046+0000","last_peered":"2023-10-21T05:33:56.709046+0000","last_clean":"2023-10-21T05:33:56.709046+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:33:56.709046+0000","last_undegraded":"2023-10-21T05:33:56.709046+0000","last_fullsized":"2023-10-21T05:33:56.709046+0000","mapping_epoch":11,"log_start":"72'2600","ondisk_log_start":"72'2600","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'8533","reported_seq":13384,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:56.692645+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:33:56.692645+0000","last_peered":"2023-10-21T05:33:56.692645+0000","last_clean":"2023-10-21T05:33:56.692645+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:33:56.692645+0000","last_undegraded":"2023-10-21T05:33:56.692645+0000","last_fullsized":"2023-10-21T05:33:56.692645+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8533,"log_dups_size":0,"ondisk_log_size":8533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'646745","reported_seq":1278595,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:55.618680+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:33:55.618680+0000","last_peered":"2023-10-21T05:33:55.618680+0000","last_clean":"2023-10-21T05:33:55.618680+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:33:55.618680+0000","last_undegraded":"2023-10-21T05:33:55.618680+0000","last_fullsized":"2023-10-21T05:33:55.618680+0000","mapping_epoch":14,"log_start":"156'636662","ondisk_log_start":"156'636662","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7791579136,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":636138,"num_read_kb":2515508,"num_write":636590,"num_write_kb":2546360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'17294","reported_seq":24899,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:55.461396+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:33:55.461396+0000","last_peered":"2023-10-21T05:33:55.461396+0000","last_clean":"2023-10-21T05:33:55.461396+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:33:55.461396+0000","last_undegraded":"2023-10-21T05:33:55.461396+0000","last_fullsized":"2023-10-21T05:33:55.461396+0000","mapping_epoch":11,"log_start":"107'7200","ondisk_log_start":"107'7200","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'654077","reported_seq":1294476,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:57.313914+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:33:57.313914+0000","last_peered":"2023-10-21T05:33:57.313914+0000","last_clean":"2023-10-21T05:33:57.313914+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:33:57.313914+0000","last_undegraded":"2023-10-21T05:33:57.313914+0000","last_fullsized":"2023-10-21T05:33:57.313914+0000","mapping_epoch":14,"log_start":"156'644061","ondisk_log_start":"156'644061","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7886770176,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":644723,"num_read_kb":2549072,"num_write":643836,"num_write_kb":2575344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3974,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:55.458834+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:33:55.458834+0000","last_peered":"2023-10-21T05:33:55.458834+0000","last_clean":"2023-10-21T05:33:55.458834+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:33:55.458834+0000","last_undegraded":"2023-10-21T05:33:55.458834+0000","last_fullsized":"2023-10-21T05:33:55.458834+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'683770","reported_seq":1349893,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:55.633906+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:33:55.633906+0000","last_peered":"2023-10-21T05:33:55.633906+0000","last_clean":"2023-10-21T05:33:55.633906+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:33:55.633906+0000","last_undegraded":"2023-10-21T05:33:55.633906+0000","last_fullsized":"2023-10-21T05:33:55.633906+0000","mapping_epoch":14,"log_start":"156'673754","ondisk_log_start":"156'673754","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8239398912,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":670634,"num_read_kb":2652388,"num_write":673056,"num_write_kb":2692224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7887,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:55.462744+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:33:55.462744+0000","last_peered":"2023-10-21T05:33:55.462744+0000","last_clean":"2023-10-21T05:33:55.462744+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:33:55.462744+0000","last_undegraded":"2023-10-21T05:33:55.462744+0000","last_fullsized":"2023-10-21T05:33:55.462744+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'628681","reported_seq":1244413,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:33:57.330743+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:33:57.330743+0000","last_peered":"2023-10-21T05:33:57.330743+0000","last_clean":"2023-10-21T05:33:57.330743+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:33:57.330743+0000","last_undegraded":"2023-10-21T05:33:57.330743+0000","last_fullsized":"2023-10-21T05:33:57.330743+0000","mapping_epoch":14,"log_start":"156'618661","ondisk_log_start":"156'618661","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":5923,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 965s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7585439763,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":619609,"num_read_kb":2450744,"num_write":618948,"num_write_kb":2475786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'8446","reported_seq":12214,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:56.711855+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:33:56.711855+0000","last_peered":"2023-10-21T05:33:56.711855+0000","last_clean":"2023-10-21T05:33:56.711855+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:33:56.711855+0000","last_undegraded":"2023-10-21T05:33:56.711855+0000","last_fullsized":"2023-10-21T05:33:56.711855+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8446,"log_dups_size":0,"ondisk_log_size":8446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":342,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:31.711284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:32:31.711284+0000","last_peered":"2023-10-21T05:32:31.711284+0000","last_clean":"2023-10-21T05:32:31.711284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:32:31.711284+0000","last_undegraded":"2023-10-21T05:32:31.711284+0000","last_fullsized":"2023-10-21T05:32:31.711284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31503187987,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2571104,"num_read_kb":10167712,"num_write":2572430,"num_write_kb":10289714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21072715776,"data_stored":40676532262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40135,"ondisk_log_size":40135,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50507,"ondisk_log_size":50507,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739502,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11521824,"kb_used_data":10289240,"kb_used_omap":1598,"kb_used_meta":1230977,"kb_avail":82850016,"statfs":{"total":96636764160,"available":84838416384,"internally_reserved":0,"allocated":10536181760,"data_stored":20337029967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1636718,"internal_metadata":1260521106},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:33:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83799999999999997,"15min":0.89000000000000001},"min":{"1min":0.28799999999999998,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":2.8399999999999999,"5min":3.7749999999999999,"15min":8.3439999999999994},"last":1.353},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.82499999999999996,"15min":0.88600000000000001},"min":{"1min":0.29099999999999998,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.8620000000000001,"5min":3.8620000000000001,"15min":5.5229999999999997},"last":0.441}]},{"osd":1,"last update":"Sat Oct 21 05:33:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81499999999999995,"15min":0.89900000000000002},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9889999999999999,"5min":3.7970000000000002,"15min":8.2710000000000008},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.92900000000000005,"15min":0.99199999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.221},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":8.3100000000000005},"last":1.609}]}]},{"osd":1,"up_from":8,"seq":34359739500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6652404,"kb_used_data":5052460,"kb_used_omap":1421,"kb_used_meta":1598514,"kb_avail":87719436,"statfs":{"total":96636764160,"available":89824702464,"internally_reserved":0,"allocated":5173719040,"data_stored":9984733987,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1636878591},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,7,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:33:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.874,"15min":0.78600000000000003},"min":{"1min":0.254,"5min":0.19,"15min":0.17199999999999999},"max":{"1min":2.867,"5min":4.6269999999999998,"15min":5.5810000000000004},"last":0.312},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.85799999999999998,"15min":0.85999999999999999},"min":{"1min":0.23000000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":2.915,"5min":4.6609999999999996,"15min":6.968},"last":1.3580000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:33:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98399999999999999,"15min":0.91700000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.20999999999999999},"max":{"1min":2.7959999999999998,"5min":5.3289999999999997,"15min":5.9950000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.84699999999999998,"15min":0.89200000000000002},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.903,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.53700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739500,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6084380,"kb_used_data":5240340,"kb_used_omap":1501,"kb_used_meta":842530,"kb_avail":88287460,"statfs":{"total":96636764160,"available":90406359040,"internally_reserved":0,"allocated":5366108160,"data_stored":10355469306,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1537489,"internal_metadata":862751279},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:33:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.78500000000000003,"15min":0.75600000000000001},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":4.4489999999999998,"5min":4.4489999999999998,"15min":5.923},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.82499999999999996,"15min":0.78500000000000003},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.23000000000000001},"max":{"1min":4.492,"5min":4.492,"15min":5.968},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:33:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.92900000000000005,"15min":1.014},"min":{"1min":0.23899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.181,"5min":7.1210000000000004,"15min":14.053000000000001},"last":8.5150000000000006},{"interface":"front","average":{"1min":0.751,"5min":1.0549999999999999,"15min":1.0349999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.1059999999999999,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.56299999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5364330496,"data_stored":10354376704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5172367360,"data_stored":9984229395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10536017920,"data_stored":20337926163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:34:00.302 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:34:00.302 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:34:11.523 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:34:11.520+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:34:11.533 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:34:11.526+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:34:14.993 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:34:14.989+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:34:16.642 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.639+0000 7f00db7b8640 -1 reset not still connected to 0x556177388b60 2023-10-21T05:34:16.643 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.639+0000 7f00db7b8640 -1 reset not still connected to 0x5561774204e0 2023-10-21T05:34:16.643 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.639+0000 7f00db7b8640 -1 reset not still connected to 0x556177421450 2023-10-21T05:34:16.649 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x5561775989c0 2023-10-21T05:34:16.650 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x556177599040 2023-10-21T05:34:16.650 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x5561775b96c0 2023-10-21T05:34:16.650 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x55617760c820 2023-10-21T05:34:16.651 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x556177af68f0 2023-10-21T05:34:16.651 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x556177b6d450 2023-10-21T05:34:16.651 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-21T05:34:16.646+0000 7f00db7b8640 -1 reset not still connected to 0x5561795e9860 2023-10-21T05:34:20.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:34:20.815 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:34:20.815 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:34:20.839 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2996,"stamp":"2023-10-21T05:34:20.072911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31540553469,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2629441,"num_read_kb":10244158,"num_write":2582579,"num_write_kb":10328091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90896,"ondisk_log_size":90896,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24658352,"kb_used_data":20655636,"kb_used_omap":4521,"kb_used_meta":3998166,"kb_avail":258457168,"statfs":{"total":289910292480,"available":264660140032,"internally_reserved":0,"allocated":21151371264,"data_stored":40752542316,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4630080,"internal_metadata":4094122432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,6,14,13,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18550784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4410,"num_read_kb":17580,"num_write":4545,"num_write_kb":18180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002087"},"pg_stats":[{"pgid":"2.7","version":"158'8516","reported_seq":12361,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:16.655441+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:34:16.655441+0000","last_peered":"2023-10-21T05:34:16.655441+0000","last_clean":"2023-10-21T05:34:16.655441+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:34:16.655441+0000","last_undegraded":"2023-10-21T05:34:16.655441+0000","last_fullsized":"2023-10-21T05:34:16.655441+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'4356","reported_seq":8329,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:15.938906+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:34:15.938906+0000","last_peered":"2023-10-21T05:34:15.938906+0000","last_clean":"2023-10-21T05:34:15.938906+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:34:15.938906+0000","last_undegraded":"2023-10-21T05:34:15.938906+0000","last_fullsized":"2023-10-21T05:34:15.938906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4356,"log_dups_size":0,"ondisk_log_size":4356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'12717","reported_seq":18336,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:16.656852+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:34:16.656852+0000","last_peered":"2023-10-21T05:34:16.656852+0000","last_clean":"2023-10-21T05:34:16.656852+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:34:16.656852+0000","last_undegraded":"2023-10-21T05:34:16.656852+0000","last_fullsized":"2023-10-21T05:34:16.656852+0000","mapping_epoch":11,"log_start":"73'2700","ondisk_log_start":"73'2700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'8561","reported_seq":13421,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:16.655589+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:34:16.655589+0000","last_peered":"2023-10-21T05:34:16.655589+0000","last_clean":"2023-10-21T05:34:16.655589+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:34:16.655589+0000","last_undegraded":"2023-10-21T05:34:16.655589+0000","last_fullsized":"2023-10-21T05:34:16.655589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8561,"log_dups_size":0,"ondisk_log_size":8561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'648853","reported_seq":1282655,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:15.788864+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:34:15.788864+0000","last_peered":"2023-10-21T05:34:15.788864+0000","last_clean":"2023-10-21T05:34:15.788864+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:34:15.788864+0000","last_undegraded":"2023-10-21T05:34:15.788864+0000","last_fullsized":"2023-10-21T05:34:15.788864+0000","mapping_epoch":14,"log_start":"157'638762","ondisk_log_start":"157'638762","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7800201216,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":638088,"num_read_kb":2523264,"num_write":638698,"num_write_kb":2554792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'17335","reported_seq":24953,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:15.039224+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:34:15.039224+0000","last_peered":"2023-10-21T05:34:15.039224+0000","last_clean":"2023-10-21T05:34:15.039224+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:34:15.039224+0000","last_undegraded":"2023-10-21T05:34:15.039224+0000","last_fullsized":"2023-10-21T05:34:15.039224+0000","mapping_epoch":11,"log_start":"109'7300","ondisk_log_start":"109'7300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'656289","reported_seq":1298934,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:17.482815+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:34:17.482815+0000","last_peered":"2023-10-21T05:34:17.482815+0000","last_clean":"2023-10-21T05:34:17.482815+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:34:17.482815+0000","last_undegraded":"2023-10-21T05:34:17.482815+0000","last_fullsized":"2023-10-21T05:34:17.482815+0000","mapping_epoch":14,"log_start":"157'646261","ondisk_log_start":"157'646261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7895830528,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":646971,"num_read_kb":2558032,"num_write":646048,"num_write_kb":2584192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3982,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:15.936949+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:34:15.936949+0000","last_peered":"2023-10-21T05:34:15.936949+0000","last_clean":"2023-10-21T05:34:15.936949+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:34:15.936949+0000","last_undegraded":"2023-10-21T05:34:15.936949+0000","last_fullsized":"2023-10-21T05:34:15.936949+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'686253","reported_seq":1354854,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:15.777456+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:34:15.777456+0000","last_peered":"2023-10-21T05:34:15.777456+0000","last_clean":"2023-10-21T05:34:15.777456+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:34:15.777456+0000","last_undegraded":"2023-10-21T05:34:15.777456+0000","last_fullsized":"2023-10-21T05:34:15.777456+0000","mapping_epoch":14,"log_start":"157'676154","ondisk_log_start":"157'676154","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8249552896,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":673109,"num_read_kb":2662244,"num_write":675539,"num_write_kb":2702156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7904,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:15.937217+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:34:15.937217+0000","last_peered":"2023-10-21T05:34:15.937217+0000","last_clean":"2023-10-21T05:34:15.937217+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:34:15.937217+0000","last_undegraded":"2023-10-21T05:34:15.937217+0000","last_fullsized":"2023-10-21T05:34:15.937217+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'630952","reported_seq":1248898,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:34:17.503698+0000","last_change":"2023-10-21T05:17:51.450259+0000","last_active":"2023-10-21T05:34:17.503698+0000","last_peered":"2023-10-21T05:34:17.503698+0000","last_clean":"2023-10-21T05:34:17.503698+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:34:17.503698+0000","last_undegraded":"2023-10-21T05:34:17.503698+0000","last_fullsized":"2023-10-21T05:34:17.503698+0000","mapping_epoch":14,"log_start":"157'620861","ondisk_log_start":"157'620861","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T03:59:50.570129+0000","last_clean_scrub_stamp":"2023-10-21T03:59:50.570129+0000","objects_scrubbed":6016,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 986s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7594373139,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":621814,"num_read_kb":2459548,"num_write":621136,"num_write_kb":2484538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'8474","reported_seq":12251,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:16.661632+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:34:16.661632+0000","last_peered":"2023-10-21T05:34:16.661632+0000","last_clean":"2023-10-21T05:34:16.661632+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:34:16.661632+0000","last_undegraded":"2023-10-21T05:34:16.661632+0000","last_fullsized":"2023-10-21T05:34:16.661632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":342,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-21T05:32:31.711284+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:32:31.711284+0000","last_peered":"2023-10-21T05:32:31.711284+0000","last_clean":"2023-10-21T05:32:31.711284+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:32:31.711284+0000","last_undegraded":"2023-10-21T05:32:31.711284+0000","last_fullsized":"2023-10-21T05:32:31.711284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31539957779,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2579982,"num_read_kb":10203088,"num_write":2581421,"num_write_kb":10325678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21146107904,"data_stored":40749867046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40309,"ondisk_log_size":40309,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50488,"ondisk_log_size":50488,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739506,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11739428,"kb_used_data":10326428,"kb_used_omap":1598,"kb_used_meta":1411393,"kb_avail":82632412,"statfs":{"total":96636764160,"available":84615589888,"internally_reserved":0,"allocated":10574262272,"data_stored":20375081807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1636718,"internal_metadata":1445267090},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:33:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83799999999999997,"15min":0.89000000000000001},"min":{"1min":0.28799999999999998,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":2.8399999999999999,"5min":3.7749999999999999,"15min":8.3439999999999994},"last":4.3079999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.82499999999999996,"15min":0.88600000000000001},"min":{"1min":0.29099999999999998,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.8620000000000001,"5min":3.8620000000000001,"15min":5.5229999999999997},"last":0.23699999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:33:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81499999999999995,"15min":0.89900000000000002},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9889999999999999,"5min":3.7970000000000002,"15min":8.2710000000000008},"last":4.3620000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.92900000000000005,"15min":0.99199999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.221},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":8.3100000000000005},"last":4.3940000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739504,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6760836,"kb_used_data":5070716,"kb_used_omap":1421,"kb_used_meta":1688690,"kb_avail":87611004,"statfs":{"total":96636764160,"available":89713668096,"internally_reserved":0,"allocated":5192413184,"data_stored":10003407651,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1729218815},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,0,1,10,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:33:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.874,"15min":0.78600000000000003},"min":{"1min":0.254,"5min":0.19,"15min":0.17199999999999999},"max":{"1min":2.867,"5min":4.6269999999999998,"15min":5.5810000000000004},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.85799999999999998,"15min":0.85999999999999999},"min":{"1min":0.23000000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":2.915,"5min":4.6609999999999996,"15min":6.968},"last":0.90800000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:33:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98399999999999999,"15min":0.91700000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.20999999999999999},"max":{"1min":2.7959999999999998,"5min":5.3289999999999997,"15min":5.9950000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.84699999999999998,"15min":0.89200000000000002},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.903,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6158088,"kb_used_data":5258492,"kb_used_omap":1501,"kb_used_meta":898082,"kb_avail":88213752,"statfs":{"total":96636764160,"available":90330882048,"internally_reserved":0,"allocated":5384695808,"data_stored":10374052858,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1537489,"internal_metadata":919636527},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:33:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.78500000000000003,"15min":0.75600000000000001},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":4.4489999999999998,"5min":4.4489999999999998,"15min":5.923},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.82499999999999996,"15min":0.78500000000000003},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.23000000000000001},"max":{"1min":4.492,"5min":4.492,"15min":5.968},"last":0.89500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:33:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.92900000000000005,"15min":1.014},"min":{"1min":0.23899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.181,"5min":7.1210000000000004,"15min":14.053000000000001},"last":0.751},{"interface":"front","average":{"1min":0.751,"5min":1.0549999999999999,"15min":1.0349999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.1059999999999999,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5383106560,"data_stored":10373144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5190340608,"data_stored":10002182163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10572660736,"data_stored":20374540307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:34:20.841 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T03:59:50.570129+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=3, tm_min=59, tm_sec=50, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=6, tm_sec=14, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:34:20.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:34:21.351 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:34:21.374 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:34:32.754 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:34:32.747+0000 7f1e40431640 -1 reset not still connected to 0x56047b722000 2023-10-21T05:34:32.755 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-10-21T05:34:32.747+0000 7f1e40431640 -1 reset not still connected to 0x56047b722270 2023-10-21T05:34:41.376 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:34:41.927 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-21T05:34:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-21T05:34:41.948 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":3006,"stamp":"2023-10-21T05:34:40.076429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31576954621,"num_objects":25489,"num_object_clones":20224,"num_object_copies":50978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25489,"num_whiteouts":0,"num_read":2638453,"num_read_kb":10280118,"num_write":2591582,"num_write_kb":10364103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90954,"ondisk_log_size":90954,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25061908,"kb_used_data":20726268,"kb_used_omap":4521,"kb_used_meta":4331094,"kb_avail":258053612,"statfs":{"total":289910292480,"available":264246898688,"internally_reserved":0,"allocated":21223698432,"data_stored":40823792236,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4630080,"internal_metadata":4435040704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,6,12,15,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18530304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4616,"num_read_kb":18428,"num_write":4587,"num_write_kb":18348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002180"},"pg_stats":[{"pgid":"2.7","version":"158'8544","reported_seq":12395,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:37.383375+0000","last_change":"2023-10-21T05:06:29.114290+0000","last_active":"2023-10-21T05:34:37.383375+0000","last_peered":"2023-10-21T05:34:37.383375+0000","last_clean":"2023-10-21T05:34:37.383375+0000","last_became_active":"2023-10-21T03:59:45.537451+0000","last_became_peered":"2023-10-21T03:59:45.537451+0000","last_unstale":"2023-10-21T05:34:37.383375+0000","last_undegraded":"2023-10-21T05:34:37.383375+0000","last_fullsized":"2023-10-21T05:34:37.383375+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6397","last_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_deep_scrub":"135'6397","last_deep_scrub_stamp":"2023-10-21T05:06:29.114059+0000","last_clean_scrub_stamp":"2023-10-21T05:06:29.114059+0000","objects_scrubbed":18,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:05:43.942007+0000","scrub_duration":0.108436647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4488,"num_read_kb":3649,"num_write":98,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'4370","reported_seq":8347,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:36.435566+0000","last_change":"2023-10-21T05:06:32.096174+0000","last_active":"2023-10-21T05:34:36.435566+0000","last_peered":"2023-10-21T05:34:36.435566+0000","last_clean":"2023-10-21T05:34:36.435566+0000","last_became_active":"2023-10-21T03:59:45.536715+0000","last_became_peered":"2023-10-21T03:59:45.536715+0000","last_unstale":"2023-10-21T05:34:36.435566+0000","last_undegraded":"2023-10-21T05:34:36.435566+0000","last_fullsized":"2023-10-21T05:34:36.435566+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'3282","last_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_deep_scrub":"135'3282","last_deep_scrub_stamp":"2023-10-21T05:06:32.095993+0000","last_clean_scrub_stamp":"2023-10-21T05:06:32.095993+0000","objects_scrubbed":18,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:03:09.564490+0000","scrub_duration":0.071696706999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5771,"num_read_kb":5315,"num_write":98,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2578,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'12759","reported_seq":18389,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:37.389414+0000","last_change":"2023-10-21T05:06:31.066792+0000","last_active":"2023-10-21T05:34:37.389414+0000","last_peered":"2023-10-21T05:34:37.389414+0000","last_clean":"2023-10-21T05:34:37.389414+0000","last_became_active":"2023-10-21T03:59:45.536904+0000","last_became_peered":"2023-10-21T03:59:45.536904+0000","last_unstale":"2023-10-21T05:34:37.389414+0000","last_undegraded":"2023-10-21T05:34:37.389414+0000","last_fullsized":"2023-10-21T05:34:37.389414+0000","mapping_epoch":11,"log_start":"73'2700","ondisk_log_start":"73'2700","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'9547","last_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_deep_scrub":"135'9547","last_deep_scrub_stamp":"2023-10-21T05:06:31.066595+0000","last_clean_scrub_stamp":"2023-10-21T05:06:31.066595+0000","objects_scrubbed":11,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:53:09.186635+0000","scrub_duration":0.070610747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":6702,"num_read_kb":5448,"num_write":125,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3507,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'8589","reported_seq":13456,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:37.382841+0000","last_change":"2023-10-21T05:06:34.111884+0000","last_active":"2023-10-21T05:34:37.382841+0000","last_peered":"2023-10-21T05:34:37.382841+0000","last_clean":"2023-10-21T05:34:37.382841+0000","last_became_active":"2023-10-21T03:59:45.536744+0000","last_became_peered":"2023-10-21T03:59:45.536744+0000","last_unstale":"2023-10-21T05:34:37.382841+0000","last_undegraded":"2023-10-21T05:34:37.382841+0000","last_fullsized":"2023-10-21T05:34:37.382841+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'6447","last_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_deep_scrub":"135'6447","last_deep_scrub_stamp":"2023-10-21T05:06:34.111663+0000","last_clean_scrub_stamp":"2023-10-21T05:06:34.111663+0000","objects_scrubbed":24,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:18:33.359172+0000","scrub_duration":0.115289981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":887,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5566,"num_read_kb":4663,"num_write":123,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2867,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'651081","reported_seq":1287095,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:35.868466+0000","last_change":"2023-10-21T05:17:46.841709+0000","last_active":"2023-10-21T05:34:35.868466+0000","last_peered":"2023-10-21T05:34:35.868466+0000","last_clean":"2023-10-21T05:34:35.868466+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:34:35.868466+0000","last_undegraded":"2023-10-21T05:34:35.868466+0000","last_fullsized":"2023-10-21T05:34:35.868466+0000","mapping_epoch":14,"log_start":"157'641062","ondisk_log_start":"157'641062","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'546678","last_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_deep_scrub":"144'546678","last_deep_scrub_stamp":"2023-10-21T05:17:46.841548+0000","last_clean_scrub_stamp":"2023-10-21T05:17:46.841548+0000","objects_scrubbed":6271,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":689,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:40:38.663939+0000","scrub_duration":688.89661666799998,"objects_trimmed":0,"snaptrim_duration":0.107393156,"stat_sum":{"num_bytes":7809179648,"num_objects":6271,"num_object_clones":5005,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":640303,"num_read_kb":2532092,"num_write":640926,"num_write_kb":2563704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'17375","reported_seq":25009,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:35.010750+0000","last_change":"2023-10-21T05:06:17.223299+0000","last_active":"2023-10-21T05:34:35.010750+0000","last_peered":"2023-10-21T05:34:35.010750+0000","last_clean":"2023-10-21T05:34:35.010750+0000","last_became_active":"2023-10-21T03:59:45.534112+0000","last_became_peered":"2023-10-21T03:59:45.534112+0000","last_unstale":"2023-10-21T05:34:35.010750+0000","last_undegraded":"2023-10-21T05:34:35.010750+0000","last_fullsized":"2023-10-21T05:34:35.010750+0000","mapping_epoch":11,"log_start":"109'7300","ondisk_log_start":"109'7300","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'12951","last_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_deep_scrub":"134'12951","last_deep_scrub_stamp":"2023-10-21T05:06:17.222911+0000","last_clean_scrub_stamp":"2023-10-21T05:06:17.222911+0000","objects_scrubbed":23,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:58:55.727157+0000","scrub_duration":0.27729903299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9094,"num_read_kb":7458,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6036,"num_omap_keys":76,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'658645","reported_seq":1303589,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:37.665817+0000","last_change":"2023-10-21T05:19:06.987027+0000","last_active":"2023-10-21T05:34:37.665817+0000","last_peered":"2023-10-21T05:34:37.665817+0000","last_clean":"2023-10-21T05:34:37.665817+0000","last_became_active":"2023-10-21T03:59:52.554918+0000","last_became_peered":"2023-10-21T03:59:52.554918+0000","last_unstale":"2023-10-21T05:34:37.665817+0000","last_undegraded":"2023-10-21T05:34:37.665817+0000","last_fullsized":"2023-10-21T05:34:37.665817+0000","mapping_epoch":14,"log_start":"157'648561","ondisk_log_start":"157'648561","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'560606","last_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_deep_scrub":"145'560606","last_deep_scrub_stamp":"2023-10-21T05:19:06.986616+0000","last_clean_scrub_stamp":"2023-10-21T05:19:06.986616+0000","objects_scrubbed":6339,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":753,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:10:13.732314+0000","scrub_duration":752.00179285599995,"objects_trimmed":0,"snaptrim_duration":0.003427041,"stat_sum":{"num_bytes":7905353728,"num_objects":6339,"num_object_clones":5057,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":649271,"num_read_kb":2567224,"num_write":648404,"num_write_kb":2593616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"153'172","reported_seq":3988,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:31.300533+0000","last_change":"2023-10-21T05:06:21.402940+0000","last_active":"2023-10-21T05:34:31.300533+0000","last_peered":"2023-10-21T05:34:31.300533+0000","last_clean":"2023-10-21T05:34:31.300533+0000","last_became_active":"2023-10-21T03:59:45.534048+0000","last_became_peered":"2023-10-21T03:59:45.534048+0000","last_unstale":"2023-10-21T05:34:31.300533+0000","last_undegraded":"2023-10-21T05:34:31.300533+0000","last_fullsized":"2023-10-21T05:34:31.300533+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'170","last_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_deep_scrub":"134'170","last_deep_scrub_stamp":"2023-10-21T05:06:21.402720+0000","last_clean_scrub_stamp":"2023-10-21T05:06:21.402720+0000","objects_scrubbed":13,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:18:00.389752+0000","scrub_duration":0.068039239000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4287,"num_read_kb":3506,"num_write":88,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2121,"num_omap_keys":27,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'688380","reported_seq":1359230,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:35.816428+0000","last_change":"2023-10-21T05:21:36.768951+0000","last_active":"2023-10-21T05:34:35.816428+0000","last_peered":"2023-10-21T05:34:35.816428+0000","last_clean":"2023-10-21T05:34:35.816428+0000","last_became_active":"2023-10-21T03:59:52.554823+0000","last_became_peered":"2023-10-21T03:59:52.554823+0000","last_unstale":"2023-10-21T05:34:35.816428+0000","last_undegraded":"2023-10-21T05:34:35.816428+0000","last_fullsized":"2023-10-21T05:34:35.816428+0000","mapping_epoch":14,"log_start":"157'678354","ondisk_log_start":"157'678354","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'601222","last_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_deep_scrub":"147'601222","last_deep_scrub_stamp":"2023-10-21T05:21:36.768717+0000","last_clean_scrub_stamp":"2023-10-21T05:21:36.768717+0000","objects_scrubbed":6632,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":916,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:44:01.488103+0000","scrub_duration":915.84441514800005,"objects_trimmed":0,"snaptrim_duration":0.00086840099999999998,"stat_sum":{"num_bytes":8258134016,"num_objects":6632,"num_object_clones":5293,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":675358,"num_read_kb":2671224,"num_write":677666,"num_write_kb":2710664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"153'357","reported_seq":7916,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:31.299007+0000","last_change":"2023-10-21T05:06:18.373978+0000","last_active":"2023-10-21T05:34:31.299007+0000","last_peered":"2023-10-21T05:34:31.299007+0000","last_clean":"2023-10-21T05:34:31.299007+0000","last_became_active":"2023-10-21T03:59:45.534029+0000","last_became_peered":"2023-10-21T03:59:45.534029+0000","last_unstale":"2023-10-21T05:34:31.299007+0000","last_undegraded":"2023-10-21T05:34:31.299007+0000","last_fullsized":"2023-10-21T05:34:31.299007+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'353","last_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_deep_scrub":"134'353","last_deep_scrub_stamp":"2023-10-21T05:06:18.373811+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.373811+0000","objects_scrubbed":14,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:09:05.610004+0000","scrub_duration":0.063528091999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9004,"num_read_kb":7332,"num_write":184,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4675,"num_omap_keys":56,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'633319","reported_seq":1253524,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:37.685151+0000","last_change":"2023-10-21T05:34:31.574406+0000","last_active":"2023-10-21T05:34:37.685151+0000","last_peered":"2023-10-21T05:34:37.685151+0000","last_clean":"2023-10-21T05:34:37.685151+0000","last_became_active":"2023-10-21T03:59:52.555285+0000","last_became_peered":"2023-10-21T03:59:52.555285+0000","last_unstale":"2023-10-21T05:34:37.685151+0000","last_undegraded":"2023-10-21T05:34:37.685151+0000","last_fullsized":"2023-10-21T05:34:37.685151+0000","mapping_epoch":14,"log_start":"157'623261","ondisk_log_start":"157'623261","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'632559","last_scrub_stamp":"2023-10-21T05:34:31.574207+0000","last_deep_scrub":"158'632559","last_deep_scrub_stamp":"2023-10-21T05:34:31.574207+0000","last_clean_scrub_stamp":"2023-10-21T05:34:31.574207+0000","objects_scrubbed":6103,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1001,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:29:58.753948+0000","scrub_duration":1000.123800579,"objects_trimmed":0,"snaptrim_duration":0.015975653999999999,"stat_sum":{"num_bytes":7603691539,"num_objects":6103,"num_object_clones":4869,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":624062,"num_read_kb":2468508,"num_write":623428,"num_write_kb":2493706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'8502","reported_seq":12285,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:34:37.388717+0000","last_change":"2023-10-21T05:06:24.088456+0000","last_active":"2023-10-21T05:34:37.388717+0000","last_peered":"2023-10-21T05:34:37.388717+0000","last_clean":"2023-10-21T05:34:37.388717+0000","last_became_active":"2023-10-21T03:59:45.536667+0000","last_became_peered":"2023-10-21T03:59:45.536667+0000","last_unstale":"2023-10-21T05:34:37.388717+0000","last_undegraded":"2023-10-21T05:34:37.388717+0000","last_fullsized":"2023-10-21T05:34:37.388717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'6355","last_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_deep_scrub":"134'6355","last_deep_scrub_stamp":"2023-10-21T05:06:24.088301+0000","last_clean_scrub_stamp":"2023-10-21T05:06:24.088301+0000","objects_scrubbed":21,"log_size":8502,"log_dups_size":0,"ondisk_log_size":8502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:10:48.219295+0000","scrub_duration":0.18905487300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4441,"num_read_kb":3608,"num_write":91,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"134'99","reported_seq":343,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-21T05:33:34.676543+0000","last_change":"2023-10-21T05:06:18.866050+0000","last_active":"2023-10-21T05:33:34.676543+0000","last_peered":"2023-10-21T05:33:34.676543+0000","last_clean":"2023-10-21T05:33:34.676543+0000","last_became_active":"2023-10-21T03:59:43.545516+0000","last_became_peered":"2023-10-21T03:59:43.545516+0000","last_unstale":"2023-10-21T05:33:34.676543+0000","last_undegraded":"2023-10-21T05:33:34.676543+0000","last_fullsized":"2023-10-21T05:33:34.676543+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'99","last_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_deep_scrub":"134'99","last_deep_scrub_stamp":"2023-10-21T05:06:18.865495+0000","last_clean_scrub_stamp":"2023-10-21T05:06:18.865495+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:29:48.202277+0000","scrub_duration":0.059192559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":31576358931,"num_objects":25345,"num_object_clones":20224,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":2588994,"num_read_kb":10239048,"num_write":2590424,"num_write_kb":10361690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21218603008,"data_stored":40821288998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40187,"ondisk_log_size":40187,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5322,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":49353,"num_read_kb":40979,"num_write":1021,"num_write_kb":519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26460,"num_omap_keys":382,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974848,"data_stored":10644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61832,"internal_metadata":0},"log_size":50668,"ondisk_log_size":50668,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6914,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739510,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11954552,"kb_used_data":10361776,"kb_used_omap":1598,"kb_used_meta":1591169,"kb_avail":82417288,"statfs":{"total":96636764160,"available":84395302912,"internally_reserved":0,"allocated":10610458624,"data_stored":20410753871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1636718,"internal_metadata":1629357714},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:33:40 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83799999999999997,"15min":0.89000000000000001},"min":{"1min":0.28799999999999998,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":2.8399999999999999,"5min":3.7749999999999999,"15min":8.3439999999999994},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.82499999999999996,"15min":0.88600000000000001},"min":{"1min":0.29099999999999998,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.8620000000000001,"5min":3.8620000000000001,"15min":5.5229999999999997},"last":0.66800000000000004}]},{"osd":1,"last update":"Sat Oct 21 05:33:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81499999999999995,"15min":0.89900000000000002},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9889999999999999,"5min":3.7970000000000002,"15min":8.2710000000000008},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.92900000000000005,"15min":0.99199999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.221},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":8.3100000000000005},"last":0.505}]}]},{"osd":1,"up_from":8,"seq":34359739508,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6879784,"kb_used_data":5088992,"kb_used_omap":1421,"kb_used_meta":1789362,"kb_avail":87492056,"statfs":{"total":96636764160,"available":89591865344,"internally_reserved":0,"allocated":5211127808,"data_stored":10021786403,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1455873,"internal_metadata":1832306943},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,7,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:33:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.874,"15min":0.78600000000000003},"min":{"1min":0.254,"5min":0.19,"15min":0.17199999999999999},"max":{"1min":2.867,"5min":4.6269999999999998,"15min":5.5810000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.85799999999999998,"15min":0.85999999999999999},"min":{"1min":0.23000000000000001,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":2.915,"5min":4.6609999999999996,"15min":6.968},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:33:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98399999999999999,"15min":0.91700000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.20999999999999999},"max":{"1min":2.7959999999999998,"5min":5.3289999999999997,"15min":5.9950000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.84699999999999998,"15min":0.89200000000000002},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.903,"5min":5.2880000000000003,"15min":5.9480000000000004},"last":2.5179999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739508,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6227572,"kb_used_data":5275500,"kb_used_omap":1501,"kb_used_meta":950562,"kb_avail":88144268,"statfs":{"total":96636764160,"available":90259730432,"internally_reserved":0,"allocated":5402112000,"data_stored":10391251962,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1537489,"internal_metadata":973376047},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,2,6,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:33:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.78500000000000003,"15min":0.75600000000000001},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":4.4489999999999998,"5min":4.4489999999999998,"15min":5.923},"last":1.2729999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.82499999999999996,"15min":0.78500000000000003},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.23000000000000001},"max":{"1min":4.492,"5min":4.492,"15min":5.968},"last":1.3089999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:33:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.92900000000000005,"15min":1.014},"min":{"1min":0.23899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.181,"5min":7.1210000000000004,"15min":14.053000000000001},"last":1.091},{"interface":"front","average":{"1min":0.751,"5min":1.0549999999999999,"15min":1.0349999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.1059999999999999,"5min":9.4469999999999992,"15min":9.4469999999999992},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3457,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19984,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":5322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30732,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11116,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5400666112,"data_stored":10390491136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5209182208,"data_stored":10020687891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10608754688,"data_stored":20410109971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:34:41.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-21T05:34:42.539 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-21T05:34:42.540 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-21T05:34:42.540 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-21T05:34:42.541 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:34:47.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-21T05:34:48.642 INFO:tasks.ceph.osd.0:Stopped 2023-10-21T05:34:48.643 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-21T05:34:48.643 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:34:52.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-21T05:34:52.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-21T05:34:54.745 INFO:tasks.ceph.osd.1:Stopped 2023-10-21T05:34:54.746 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-21T05:34:54.746 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:34:57.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-21T05:34:57.763 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-21T05:35:00.848 INFO:tasks.ceph.osd.2:Stopped 2023-10-21T05:35:00.849 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-21T05:35:00.849 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-21T05:35:00.850 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:35:00.880 INFO:tasks.ceph.mgr.x:Stopped 2023-10-21T05:35:00.880 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-21T05:35:00.881 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-21T05:35:00.881 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:35:00.939 INFO:tasks.ceph.mon.a:Stopped 2023-10-21T05:35:00.939 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-21T05:35:00.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-21T05:35:01.071 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T05:35:01.071 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-21T05:35:02.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-21T05:35:03.966 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T05:35:03.967 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-21T05:35:04.347 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T05:35:04.347 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-21T05:35:05.113 INFO:tasks.ceph:Archiving mon data... 2023-10-21T05:35:05.114 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357/data/mon.a.tgz 2023-10-21T05:35:05.115 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-10-21T05:35:05.139 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.iSuUd3kqxN 2023-10-21T05:35:05.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iSuUd3kqxN 2023-10-21T05:35:05.323 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.iSuUd3kqxN 2023-10-21T05:35:05.416 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.iSuUd3kqxN is 626KB 2023-10-21T05:35:05.489 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.iSuUd3kqxN 2023-10-21T05:35:05.512 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-21T05:35:05.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-21T05:35:05.602 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-21T05:35:05.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-21T05:35:05.665 INFO:teuthology.orchestra.run.smithi040.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-21T05:35:05.670 INFO:tasks.ceph:Archiving crash dumps... 2023-10-21T05:35:05.672 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357/remote/smithi040/crash 2023-10-21T05:35:05.673 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-21T05:35:05.706 INFO:tasks.ceph:Compressing logs... 2023-10-21T05:35:05.707 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-21T05:35:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103124.log 2023-10-21T05:35:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89673.log 2023-10-21T05:35:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.103124.log: /var/log/ceph/ceph-client.admin.105327.log 2023-10-21T05:35:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92851.log 2023-10-21T05:35:05.853 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89673.log.gz 2023-10-21T05:35:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103034.log 2023-10-21T05:35:05.854 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105327.log.gz 2023-10-21T05:35:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105832.log 2023-10-21T05:35:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94244.log 2023-10-21T05:35:05.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95195.log 2023-10-21T05:35:05.855 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105832.log.gz 2023-10-21T05:35:05.855 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88718.log 2023-10-21T05:35:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94244.log: /var/log/ceph/ceph-client.admin.95195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102851.log 2023-10-21T05:35:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106810.log 2023-10-21T05:35:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100985.log 2023-10-21T05:35:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84303.log 2023-10-21T05:35:06.038 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106810.log.gz 2023-10-21T05:35:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104775.log 2023-10-21T05:35:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.84303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96086.log 2023-10-21T05:35:06.039 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84303.log.gz 2023-10-21T05:35:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97495.log 2023-10-21T05:35:06.039 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104775.log.gz 2023-10-21T05:35:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105748.log 2023-10-21T05:35:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97495.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-21T05:35:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101680.log 2023-10-21T05:35:06.040 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105748.log.gz 2023-10-21T05:35:06.041 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89315.log 2023-10-21T05:35:06.041 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88755.log 2023-10-21T05:35:06.041 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96209.log 2023-10-21T05:35:06.041 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93570.log 2023-10-21T05:35:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98323.log 2023-10-21T05:35:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101955.log 2023-10-21T05:35:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103178.log 2023-10-21T05:35:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97067.log 2023-10-21T05:35:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92493.log 2023-10-21T05:35:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91266.log 2023-10-21T05:35:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88954.log 2023-10-21T05:35:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107065.log 2023-10-21T05:35:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102411.log 2023-10-21T05:35:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84505.log 2023-10-21T05:35:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103015.log 2023-10-21T05:35:06.045 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106430.log 2023-10-21T05:35:06.045 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89133.log 2023-10-21T05:35:06.045 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102503.log 2023-10-21T05:35:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100804.log 2023-10-21T05:35:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103795.log 2023-10-21T05:35:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104220.log 2023-10-21T05:35:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89099.log 2023-10-21T05:35:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100864.log 2023-10-21T05:35:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93912.log 2023-10-21T05:35:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94981.log 2023-10-21T05:35:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94157.log 2023-10-21T05:35:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92035.log 2023-10-21T05:35:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100620.log 2023-10-21T05:35:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102473.log 2023-10-21T05:35:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89187.log 2023-10-21T05:35:06.049 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88775.log 2023-10-21T05:35:06.049 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95811.log 2023-10-21T05:35:06.049 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108178.log 2023-10-21T05:35:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93570.log: /var/log/ceph/ceph-client.admin.98323.log: /var/log/ceph/ceph-client.admin.101955.log: /var/log/ceph/ceph-client.admin.103178.log: /var/log/ceph/ceph-client.admin.97067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93570.log.gz 2023-10-21T05:35:06.050 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98323.log.gz 2023-10-21T05:35:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92493.log: /var/log/ceph/ceph-client.admin.95811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103178.log.gz 2023-10-21T05:35:06.051 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101955.log.gz 2023-10-21T05:35:06.051 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92493.log.gz 2023-10-21T05:35:06.051 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95811.log.gz 2023-10-21T05:35:06.051 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98363.log 2023-10-21T05:35:06.052 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97067.log.gz 2023-10-21T05:35:06.052 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91266.log: /var/log/ceph/ceph-client.admin.88954.log: /var/log/ceph/ceph-client.admin.107065.log: /var/log/ceph/ceph-client.admin.102411.log: /var/log/ceph/ceph-client.admin.84505.log: /var/log/ceph/ceph-client.admin.108178.log: /var/log/ceph/ceph-client.admin.103015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107065.log.gz 2023-10-21T05:35:06.052 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102411.log.gz 2023-10-21T05:35:06.052 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84505.log.gzgzip 2023-10-21T05:35:06.053 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92606.log 2023-10-21T05:35:06.053 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108178.log.gz 2023-10-21T05:35:06.053 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103015.log.gz 2023-10-21T05:35:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106430.log: /var/log/ceph/ceph-client.admin.89133.log: /var/log/ceph/ceph-client.admin.102503.log: /var/log/ceph/ceph-client.admin.100804.log: /var/log/ceph/ceph-client.admin.103795.log: /var/log/ceph/ceph-client.admin.98363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102503.log.gz 0.0% 2023-10-21T05:35:06.054 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106430.log.gz 2023-10-21T05:35:06.054 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100804.log.gz 2023-10-21T05:35:06.055 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103795.log.gz 2023-10-21T05:35:06.055 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98363.log.gz 2023-10-21T05:35:06.055 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101468.log 2023-10-21T05:35:06.055 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104220.log: /var/log/ceph/ceph-client.admin.89099.log: /var/log/ceph/ceph-client.admin.100864.log: /var/log/ceph/ceph-client.admin.93912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104220.log.gz 2023-10-21T05:35:06.056 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100864.log.gz 2023-10-21T05:35:06.056 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93912.log.gz 2023-10-21T05:35:06.056 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0%/var/log/ceph/ceph-client.admin.94157.log: -- replaced with /var/log/ceph/ceph-client.admin.94981.log.gz 2023-10-21T05:35:06.056 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89278.log 2023-10-21T05:35:06.057 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92606.log.gz 2023-10-21T05:35:06.057 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100620.log: /var/log/ceph/ceph-client.admin.102473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94157.log.gz 2023-10-21T05:35:06.057 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97495.log.gz 2023-10-21T05:35:06.057 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102851.log.gz 2023-10-21T05:35:06.058 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100620.log.gz 2023-10-21T05:35:06.058 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101680.log.gz 2023-10-21T05:35:06.058 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100985.log.gz 2023-10-21T05:35:06.058 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.102473.log.gz 2023-10-21T05:35:06.059 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103034.log.gz 2023-10-21T05:35:06.059 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103124.log.gz 2023-10-21T05:35:06.059 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92851.log.gz 2023-10-21T05:35:06.060 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101468.log.gz 2023-10-21T05:35:06.060 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94244.log.gz 2023-10-21T05:35:06.060 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96209.log.gz 2023-10-21T05:35:06.060 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96086.log.gz 2023-10-21T05:35:06.061 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95195.log.gz 2023-10-21T05:35:06.061 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92188.log 2023-10-21T05:35:06.061 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88937.log 2023-10-21T05:35:06.061 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88917.log 2023-10-21T05:35:06.062 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101528.log 2023-10-21T05:35:06.062 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104732.log 2023-10-21T05:35:06.062 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104604.log 2023-10-21T05:35:06.063 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101528.log.gz 2023-10-21T05:35:06.063 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99768.log 2023-10-21T05:35:06.063 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104732.log.gz 2023-10-21T05:35:06.064 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93233.log 2023-10-21T05:35:06.064 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104604.log.gz 2023-10-21T05:35:06.064 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102887.log 2023-10-21T05:35:06.064 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99768.log.gz 2023-10-21T05:35:06.065 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92523.log 2023-10-21T05:35:06.065 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93233.log.gz 2023-10-21T05:35:06.065 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105706.log 2023-10-21T05:35:06.066 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102887.log.gz 2023-10-21T05:35:06.066 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108220.log 2023-10-21T05:35:06.066 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92523.log.gz 2023-10-21T05:35:06.067 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107366.log 2023-10-21T05:35:06.067 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105706.log.gz 2023-10-21T05:35:06.067 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.108220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102707.log 2023-10-21T05:35:06.067 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108220.log.gz 2023-10-21T05:35:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.107366.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.107108.log 2023-10-21T05:35:06.068 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107366.log.gz 2023-10-21T05:35:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87626.log 2023-10-21T05:35:06.068 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102707.log.gz 2023-10-21T05:35:06.069 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96674.log 2023-10-21T05:35:06.069 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107108.log.gz 2023-10-21T05:35:06.069 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96026.log 2023-10-21T05:35:06.070 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87626.log.gz 2023-10-21T05:35:06.070 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104262.log 2023-10-21T05:35:06.070 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96674.log.gz 2023-10-21T05:35:06.070 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95316.log 2023-10-21T05:35:06.071 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96026.log.gz 2023-10-21T05:35:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99982.log 2023-10-21T05:35:06.071 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104262.log.gz 2023-10-21T05:35:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101590.log 2023-10-21T05:35:06.072 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91266.log.gz 2023-10-21T05:35:06.072 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92035.log.gz 2023-10-21T05:35:06.072 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92188.log.gz 0.0% 2023-10-21T05:35:06.073 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95316.log.gz 2023-10-21T05:35:06.073 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101166.log 2023-10-21T05:35:06.073 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99982.log.gz 2023-10-21T05:35:06.073 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108092.log 2023-10-21T05:35:06.074 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101590.log.gz 2023-10-21T05:35:06.074 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88792.log 2023-10-21T05:35:06.074 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101166.log.gz 2023-10-21T05:35:06.074 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.108092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89440.log 2023-10-21T05:35:06.075 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108092.log.gz 2023-10-21T05:35:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100834.log 2023-10-21T05:35:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89548.log 2023-10-21T05:35:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89440.log.gz 2023-10-21T05:35:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96330.log 2023-10-21T05:35:06.076 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88775.log.gz 2023-10-21T05:35:06.076 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88718.log.gz 2023-10-21T05:35:06.077 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88792.log.gz 2023-10-21T05:35:06.077 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88755.log.gz 2023-10-21T05:35:06.077 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100834.log.gz 2023-10-21T05:35:06.078 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.89548.log: -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88954.log.gz 2023-10-21T05:35:06.078 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89187.log.gz 0.0% 2023-10-21T05:35:06.078 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89315.log.gz 2023-10-21T05:35:06.078 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89099.log.gz 2023-10-21T05:35:06.079 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89548.log.gz -- replaced with /var/log/ceph/ceph-client.admin.89133.log.gz 2023-10-21T05:35:06.079 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-21T05:35:06.079 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101895.log 2023-10-21T05:35:06.080 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88917.log.gz 2023-10-21T05:35:06.080 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89278.log.gz 2023-10-21T05:35:06.080 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88937.log.gz 2023-10-21T05:35:06.080 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96330.log.gz 2023-10-21T05:35:06.081 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98203.log 2023-10-21T05:35:06.081 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.101895.log: -- /var/log/ceph/ceph-client.admin.89639.log 2023-10-21T05:35:06.081 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101895.log.gz 2023-10-21T05:35:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104989.log 2023-10-21T05:35:06.082 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98203.log.gz 2023-10-21T05:35:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107237.log 2023-10-21T05:35:06.082 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89639.log.gz 2023-10-21T05:35:06.083 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99658.log 2023-10-21T05:35:06.083 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104989.log.gz 2023-10-21T05:35:06.083 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104863.log 2023-10-21T05:35:06.083 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107237.log.gz 2023-10-21T05:35:06.084 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94275.log 2023-10-21T05:35:06.084 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99658.log.gz 2023-10-21T05:35:06.084 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107965.log 2023-10-21T05:35:06.085 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104863.log.gz 2023-10-21T05:35:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97220.log 2023-10-21T05:35:06.085 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94275.log.gz 2023-10-21T05:35:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100012.log 2023-10-21T05:35:06.086 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107965.log.gz 2023-10-21T05:35:06.086 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102814.log 2023-10-21T05:35:06.086 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97220.log.gz 2023-10-21T05:35:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97007.log 2023-10-21T05:35:06.087 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100012.log.gz 2023-10-21T05:35:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97193.log 2023-10-21T05:35:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102814.log.gz 2023-10-21T05:35:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.97007.log: --verbose -- /var/log/ceph/ceph-client.admin.96648.log 2023-10-21T05:35:06.088 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97007.log.gz 2023-10-21T05:35:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100924.log 2023-10-21T05:35:06.088 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97193.log.gz 2023-10-21T05:35:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104646.log 2023-10-21T05:35:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96648.log.gz 2023-10-21T05:35:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96158.log 2023-10-21T05:35:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100924.log.gz 2023-10-21T05:35:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101287.log 2023-10-21T05:35:06.090 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104646.log.gz 2023-10-21T05:35:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104520.log 2023-10-21T05:35:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96158.log.gz 2023-10-21T05:35:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106132.log 2023-10-21T05:35:06.091 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101287.log.gz 2023-10-21T05:35:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96545.log 2023-10-21T05:35:06.092 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104520.log.gz 2023-10-21T05:35:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.106132.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.103233.log 2023-10-21T05:35:06.092 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106132.log.gz 2023-10-21T05:35:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90721.log 2023-10-21T05:35:06.093 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96545.log.gz 2023-10-21T05:35:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95639.log 2023-10-21T05:35:06.093 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103233.log.gz 2023-10-21T05:35:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101865.log 2023-10-21T05:35:06.094 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90721.log.gz 2023-10-21T05:35:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94105.log 2023-10-21T05:35:06.094 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95639.log.gz 2023-10-21T05:35:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92278.log 2023-10-21T05:35:06.095 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101865.log.gz 2023-10-21T05:35:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98507.log 2023-10-21T05:35:06.096 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94105.log.gz 2023-10-21T05:35:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96056.log 2023-10-21T05:35:06.096 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92278.log.gz 2023-10-21T05:35:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100650.log 2023-10-21T05:35:06.097 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98507.log.gz 2023-10-21T05:35:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.96056.log: /var/log/ceph/ceph-client.admin.106642.log 2023-10-21T05:35:06.097 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96056.log.gz 2023-10-21T05:35:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102138.log 2023-10-21T05:35:06.098 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100650.log.gz 2023-10-21T05:35:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100134.log 2023-10-21T05:35:06.098 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106642.log.gz 2023-10-21T05:35:06.099 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89369.log 2023-10-21T05:35:06.099 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102138.log.gz 2023-10-21T05:35:06.099 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88006.log 2023-10-21T05:35:06.099 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100134.log.gz 2023-10-21T05:35:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104905.log 2023-10-21T05:35:06.100 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89369.log.gz 2023-10-21T05:35:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87964.log 2023-10-21T05:35:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88006.log.gz 2023-10-21T05:35:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91341.log 2023-10-21T05:35:06.101 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104905.log.gz 2023-10-21T05:35:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94401.log 2023-10-21T05:35:06.102 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87964.log.gz 2023-10-21T05:35:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97386.log 2023-10-21T05:35:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91341.log.gz 2023-10-21T05:35:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95042.log 2023-10-21T05:35:06.103 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94401.log.gz 2023-10-21T05:35:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97386.log.gz 2023-10-21T05:35:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90850.log 2023-10-21T05:35:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89810.log 2023-10-21T05:35:06.104 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95042.log.gz 2023-10-21T05:35:06.104 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91913.log 2023-10-21T05:35:06.104 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90850.log.gz 2023-10-21T05:35:06.104 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94349.log 2023-10-21T05:35:06.105 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89810.log.gz 2023-10-21T05:35:06.105 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105073.log 2023-10-21T05:35:06.105 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91913.log.gz 2023-10-21T05:35:06.105 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101711.log 2023-10-21T05:35:06.106 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94349.log.gz 2023-10-21T05:35:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103197.log 2023-10-21T05:35:06.106 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105073.log.gz 2023-10-21T05:35:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99710.log 2023-10-21T05:35:06.152 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101711.log.gz 2023-10-21T05:35:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96182.log 2023-10-21T05:35:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103197.log.gz 2023-10-21T05:35:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98263.log 2023-10-21T05:35:06.154 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99710.log.gz 2023-10-21T05:35:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100590.log 2023-10-21T05:35:06.155 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96182.log.gz 2023-10-21T05:35:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100680.log 2023-10-21T05:35:06.155 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98263.log.gz 2023-10-21T05:35:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108007.log 2023-10-21T05:35:06.156 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100590.log.gz 2023-10-21T05:35:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92960.log 2023-10-21T05:35:06.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100680.log.gz 2023-10-21T05:35:06.157 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93615.log 2023-10-21T05:35:06.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.108007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108007.log.gz 2023-10-21T05:35:06.158 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99001.log 2023-10-21T05:35:06.158 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92960.log.gz 2023-10-21T05:35:06.158 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106514.log 2023-10-21T05:35:06.159 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93615.log.gz 2023-10-21T05:35:06.159 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.99001.log: -- /var/log/ceph/ceph-client.admin.95936.log 2023-10-21T05:35:06.159 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99001.log.gz 2023-10-21T05:35:06.159 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102260.log 2023-10-21T05:35:06.160 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99686.log 2023-10-21T05:35:06.160 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95936.log.gz 2023-10-21T05:35:06.160 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104690.log 2023-10-21T05:35:06.161 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102260.log.gz 2023-10-21T05:35:06.161 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97281.log 2023-10-21T05:35:06.161 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99686.log.gz 2023-10-21T05:35:06.162 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94428.log 2023-10-21T05:35:06.162 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104690.log.gz 2023-10-21T05:35:06.162 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107408.log 2023-10-21T05:35:06.163 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97281.log.gz 2023-10-21T05:35:06.163 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94552.log 2023-10-21T05:35:06.163 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94428.log.gz 2023-10-21T05:35:06.163 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98814.log 2023-10-21T05:35:06.164 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107408.log.gz 2023-10-21T05:35:06.164 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104010.log 2023-10-21T05:35:06.164 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94552.log.gz 2023-10-21T05:35:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98019.log 2023-10-21T05:35:06.165 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98814.log.gz 2023-10-21T05:35:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89531.log 2023-10-21T05:35:06.166 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104010.log.gz 2023-10-21T05:35:06.166 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92075.log 2023-10-21T05:35:06.166 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98019.log.gz 2023-10-21T05:35:06.166 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89747.log 2023-10-21T05:35:06.167 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89531.log.gz 2023-10-21T05:35:06.167 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101742.log 2023-10-21T05:35:06.168 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92075.log.gz 2023-10-21T05:35:06.168 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93264.log 2023-10-21T05:35:06.168 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89747.log.gz 2023-10-21T05:35:06.169 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91974.log 2023-10-21T05:35:06.169 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101742.log.gz 2023-10-21T05:35:06.169 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96947.log 2023-10-21T05:35:06.169 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93264.log.gz 2023-10-21T05:35:06.170 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.91974.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.98446.log 2023-10-21T05:35:06.170 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91974.log.gz 2023-10-21T05:35:06.170 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96947.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.96947.log.gz -5 2023-10-21T05:35:06.170 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101559.log 2023-10-21T05:35:06.171 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.98446.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.100437.log 2023-10-21T05:35:06.171 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98446.log.gz 2023-10-21T05:35:06.171 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105031.log 2023-10-21T05:35:06.172 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101559.log.gz 2023-10-21T05:35:06.172 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100437.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.100437.log.gz /var/log/ceph/ceph-client.admin.104436.log 2023-10-21T05:35:06.172 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-21T05:35:06.172 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106726.log 2023-10-21T05:35:06.173 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105031.log.gz 2023-10-21T05:35:06.173 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93509.log 2023-10-21T05:35:06.173 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104436.log.gz 2023-10-21T05:35:06.174 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95255.log 2023-10-21T05:35:06.174 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106726.log.gz 2023-10-21T05:35:06.174 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105242.log 2023-10-21T05:35:06.174 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93509.log.gz 2023-10-21T05:35:06.175 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100164.log 2023-10-21T05:35:06.175 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95255.log.gz 2023-10-21T05:35:06.175 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89693.log 2023-10-21T05:35:06.176 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105242.log.gz 2023-10-21T05:35:06.176 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103498.log 2023-10-21T05:35:06.176 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93448.log 2023-10-21T05:35:06.176 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108306.log 2023-10-21T05:35:06.177 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100377.log 2023-10-21T05:35:06.177 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93294.log 2023-10-21T05:35:06.177 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95505.log 2023-10-21T05:35:06.178 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89241.log 2023-10-21T05:35:06.178 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100407.log 2023-10-21T05:35:06.178 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100044.log 2023-10-21T05:35:06.179 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98598.log 2023-10-21T05:35:06.179 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96427.log 2023-10-21T05:35:06.179 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106684.log 2023-10-21T05:35:06.180 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92005.log 2023-10-21T05:35:06.180 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98671.log 2023-10-21T05:35:06.180 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93369.log 2023-10-21T05:35:06.180 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92308.log 2023-10-21T05:35:06.181 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93860.log 2023-10-21T05:35:06.181 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108430.log 2023-10-21T05:35:06.181 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104817.log 2023-10-21T05:35:06.182 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100316.log 2023-10-21T05:35:06.182 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90901.log 2023-10-21T05:35:06.182 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90259.log 2023-10-21T05:35:06.182 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102996.log 2023-10-21T05:35:06.183 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93540.log 2023-10-21T05:35:06.183 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89766.log 2023-10-21T05:35:06.183 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97802.log 2023-10-21T05:35:06.183 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97438.log 2023-10-21T05:35:06.184 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98233.log 2023-10-21T05:35:06.184 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97741.log 2023-10-21T05:35:06.184 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96269.log 2023-10-21T05:35:06.184 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87751.log 2023-10-21T05:35:06.185 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107450.log 2023-10-21T05:35:06.185 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98152.log 2023-10-21T05:35:06.185 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97586.log 2023-10-21T05:35:06.186 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95885.log 2023-10-21T05:35:06.186 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88900.log 2023-10-21T05:35:06.186 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104052.log 2023-10-21T05:35:06.186 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105369.log 2023-10-21T05:35:06.187 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108263.log 2023-10-21T05:35:06.187 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97989.log 2023-10-21T05:35:06.187 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99489.log 2023-10-21T05:35:06.188 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89727.log 2023-10-21T05:35:06.188 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88863.log 2023-10-21T05:35:06.188 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94184.log 2023-10-21T05:35:06.188 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103925.log 2023-10-21T05:35:06.189 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108348.log 2023-10-21T05:35:06.189 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89295.log 2023-10-21T05:35:06.189 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100164.log: /var/log/ceph/ceph-client.admin.103925.log: /var/log/ceph/ceph-client.admin.89693.log: /var/log/ceph/ceph-client.admin.103498.log: /var/log/ceph/ceph-client.admin.93448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103925.log.gz 2023-10-21T05:35:06.189 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89693.log.gz 2023-10-21T05:35:06.190 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.108306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100164.log.gz -- replaced with /var/log/ceph/ceph-client.admin.103498.log.gz 2023-10-21T05:35:06.190 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-21T05:35:06.190 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93448.log.gz 2023-10-21T05:35:06.190 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100377.log: /var/log/ceph/ceph-client.admin.108348.log: /var/log/ceph/ceph-client.admin.93294.log: /var/log/ceph/ceph-client.admin.95505.log: /var/log/ceph/ceph-client.admin.89241.log: gzip/var/log/ceph/ceph-client.admin.100407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100377.log.gz 2023-10-21T05:35:06.191 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95505.log.gz 2023-10-21T05:35:06.191 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93294.log.gz 2023-10-21T05:35:06.191 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89241.log.gz 2023-10-21T05:35:06.192 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100407.log.gz 2023-10-21T05:35:06.192 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.108348.log.gz 2023-10-21T05:35:06.193 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101377.log 2023-10-21T05:35:06.193 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100044.log: /var/log/ceph/ceph-client.admin.98598.log: /var/log/ceph/ceph-client.admin.96427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108306.log.gz 2023-10-21T05:35:06.194 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100044.log.gz 2023-10-21T05:35:06.194 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106684.log: /var/log/ceph/ceph-client.admin.89295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98598.log.gz 2023-10-21T05:35:06.195 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96427.log.gz 2023-10-21T05:35:06.195 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106684.log.gz 2023-10-21T05:35:06.195 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91420.log 0.0% 2023-10-21T05:35:06.195 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89295.log.gz 2023-10-21T05:35:06.196 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98671.log: /var/log/ceph/ceph-client.admin.93369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92005.log.gz 2023-10-21T05:35:06.196 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98671.log.gz 2023-10-21T05:35:06.196 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93860.log: /var/log/ceph/ceph-client.admin.108430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93369.log.gz 2023-10-21T05:35:06.197 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92308.log.gz 2023-10-21T05:35:06.197 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93860.log.gz 2023-10-21T05:35:06.198 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0%/var/log/ceph/ceph-client.admin.100316.log: -- replaced with /var/log/ceph/ceph-client.admin.101377.log.gz 2023-10-21T05:35:06.198 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104817.log.gz/var/log/ceph/ceph-client.admin.90901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108430.log.gz 2023-10-21T05:35:06.198 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90259.log: 2023-10-21T05:35:06.199 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.100316.log.gz /var/log/ceph/ceph-client.admin.106346.log 2023-10-21T05:35:06.199 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-21T05:35:06.199 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90901.log.gz 2023-10-21T05:35:06.199 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102996.log: /var/log/ceph/ceph-client.admin.93540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90259.log.gz 2023-10-21T05:35:06.200 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89766.log: /var/log/ceph/ceph-client.admin.97802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102996.log.gz 2023-10-21T05:35:06.200 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93540.log.gz 2023-10-21T05:35:06.200 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89766.log.gz 2023-10-21T05:35:06.200 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97802.log.gz 2023-10-21T05:35:06.201 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97438.log.gz 2023-10-21T05:35:06.201 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91420.log.gz 2023-10-21T05:35:06.201 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98233.log.gz 2023-10-21T05:35:06.201 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97741.log.gz 2023-10-21T05:35:06.202 INFO:teuthology.orchestra.run.smithi040.stderr: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.106514.log.gz 2023-10-21T05:35:06.202 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87751.log: /var/log/ceph/ceph-client.admin.107450.log: /var/log/ceph/ceph-client.admin.98152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95168.log 2023-10-21T05:35:06.202 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87751.log.gz 2023-10-21T05:35:06.203 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97586.log: /var/log/ceph/ceph-client.admin.95885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96269.log.gz 2023-10-21T05:35:06.203 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98152.log.gz 2023-10-21T05:35:06.203 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97586.log.gz 2023-10-21T05:35:06.203 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95885.log.gz 2023-10-21T05:35:06.204 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88900.log: /var/log/ceph/ceph-client.admin.104052.log: /var/log/ceph/ceph-client.admin.105369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107450.log.gz 2023-10-21T05:35:06.204 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106346.log: /var/log/ceph/ceph-client.admin.108263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88900.log.gz 2023-10-21T05:35:06.204 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104052.log.gz 2023-10-21T05:35:06.205 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106346.log.gz 2023-10-21T05:35:06.205 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108263.log.gz 2023-10-21T05:35:06.205 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97989.log.gz 2023-10-21T05:35:06.206 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89727.log: /var/log/ceph/ceph-client.admin.88863.log: /var/log/ceph/ceph-client.admin.94184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93940.log 2023-10-21T05:35:06.206 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99489.log.gz -- replaced with /var/log/ceph/ceph-client.admin.89727.log.gz 2023-10-21T05:35:06.206 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-21T05:35:06.206 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88863.log.gz 2023-10-21T05:35:06.207 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94184.log.gz 2023-10-21T05:35:06.207 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84442.log 2023-10-21T05:35:06.207 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95168.log.gz 2023-10-21T05:35:06.207 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104178.log 2023-10-21T05:35:06.208 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93940.log.gz 2023-10-21T05:35:06.208 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.84442.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-21T05:35:06.209 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84442.log.gz 2023-10-21T05:35:06.209 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108390.log 2023-10-21T05:35:06.209 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104178.log.gz 2023-10-21T05:35:06.210 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101438.log 2023-10-21T05:35:06.210 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103372.log 2023-10-21T05:35:06.211 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.108390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108390.log.gz 2023-10-21T05:35:06.211 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93173.log 2023-10-21T05:35:06.211 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101438.log.gz 2023-10-21T05:35:06.212 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102832.log 2023-10-21T05:35:06.212 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103372.log.gz 2023-10-21T05:35:06.212 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91862.log 2023-10-21T05:35:06.213 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93173.log.gz 2023-10-21T05:35:06.213 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102978.log 2023-10-21T05:35:06.213 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102832.log.gz 2023-10-21T05:35:06.213 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107793.log 2023-10-21T05:35:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91862.log.gz 2023-10-21T05:35:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99305.log 2023-10-21T05:35:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102978.log.gz 2023-10-21T05:35:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93479.log 2023-10-21T05:35:06.215 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107793.log.gz 2023-10-21T05:35:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102533.log 2023-10-21T05:35:06.215 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99305.log.gz 2023-10-21T05:35:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103106.log 2023-10-21T05:35:06.216 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93479.log.gz 2023-10-21T05:35:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98752.log 2023-10-21T05:35:06.217 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102533.log.gz 2023-10-21T05:35:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90437.log 2023-10-21T05:35:06.217 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103106.log.gz 2023-10-21T05:35:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97709.log 2023-10-21T05:35:06.218 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98752.log.gz 2023-10-21T05:35:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89045.log 2023-10-21T05:35:06.218 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90437.log.gz 2023-10-21T05:35:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102077.log 2023-10-21T05:35:06.219 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97709.log.gz 2023-10-21T05:35:06.219 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88809.log 2023-10-21T05:35:06.219 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89045.log.gz 2023-10-21T05:35:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101226.log 2023-10-21T05:35:06.220 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102077.log.gz 2023-10-21T05:35:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102905.log 2023-10-21T05:35:06.221 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88809.log.gz 2023-10-21T05:35:06.221 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94521.log 2023-10-21T05:35:06.222 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101226.log.gz 2023-10-21T05:35:06.222 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108049.log 2023-10-21T05:35:06.222 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102905.log.gz 2023-10-21T05:35:06.223 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97878.log 2023-10-21T05:35:06.223 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94521.log.gz 2023-10-21T05:35:06.223 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.108049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90961.log 2023-10-21T05:35:06.224 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108049.log.gz 2023-10-21T05:35:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97465.log 2023-10-21T05:35:06.224 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97878.log.gz 2023-10-21T05:35:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95475.log 2023-10-21T05:35:06.225 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90961.log.gz 2023-10-21T05:35:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101408.log 2023-10-21T05:35:06.225 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97465.log.gz 2023-10-21T05:35:06.226 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.95475.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.99091.log 2023-10-21T05:35:06.226 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95475.log.gz 2023-10-21T05:35:06.226 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93094.log 2023-10-21T05:35:06.227 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101408.log.gz 2023-10-21T05:35:06.227 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105285.log 2023-10-21T05:35:06.227 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99091.log.gz 2023-10-21T05:35:06.227 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100774.log 2023-10-21T05:35:06.228 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93094.log.gz 2023-10-21T05:35:06.228 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94133.log 2023-10-21T05:35:06.228 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105285.log.gz 2023-10-21T05:35:06.228 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93020.log 2023-10-21T05:35:06.229 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100774.log.gz 2023-10-21T05:35:06.229 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89403.log 2023-10-21T05:35:06.229 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94133.log.gz 2023-10-21T05:35:06.230 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93020.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.92248.log 2023-10-21T05:35:06.230 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93020.log.gz 2023-10-21T05:35:06.230 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99408.log 2023-10-21T05:35:06.230 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89403.log.gz 2023-10-21T05:35:06.231 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.92248.log: --verbose -- /var/log/ceph/ceph-client.admin.87500.log 2023-10-21T05:35:06.231 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92248.log.gz 2023-10-21T05:35:06.231 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93815.log 2023-10-21T05:35:06.231 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99408.log.gz 2023-10-21T05:35:06.232 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99609.log 2023-10-21T05:35:06.232 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87500.log.gz 2023-10-21T05:35:06.232 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95909.log 2023-10-21T05:35:06.233 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93815.log.gz 2023-10-21T05:35:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104562.log 2023-10-21T05:35:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99609.log.gz 2023-10-21T05:35:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101136.log/var/log/ceph/ceph-client.admin.95909.log: 2023-10-21T05:35:06.234 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95909.log.gz 2023-10-21T05:35:06.234 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93785.log 2023-10-21T05:35:06.234 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104562.log.gz 2023-10-21T05:35:06.235 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99931.log 2023-10-21T05:35:06.235 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101136.log.gz 2023-10-21T05:35:06.235 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98050.log 2023-10-21T05:35:06.235 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93785.log.gz 2023-10-21T05:35:06.236 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107150.log/var/log/ceph/ceph-client.admin.99931.log: 2023-10-21T05:35:06.236 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99931.log.gz 2023-10-21T05:35:06.236 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96895.log 2023-10-21T05:35:06.237 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98050.log.gz 2023-10-21T05:35:06.237 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104136.log 2023-10-21T05:35:06.237 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107150.log.gz 2023-10-21T05:35:06.238 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98538.log 2023-10-21T05:35:06.238 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96895.log.gz 2023-10-21T05:35:06.238 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90669.log 2023-10-21T05:35:06.238 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104136.log.gz 2023-10-21T05:35:06.239 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.98538.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.92432.log 2023-10-21T05:35:06.239 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98538.log.gz 2023-10-21T05:35:06.239 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106304.log 2023-10-21T05:35:06.240 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90669.log.gz 2023-10-21T05:35:06.240 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95012.log 2023-10-21T05:35:06.240 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92432.log.gz 2023-10-21T05:35:06.240 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98080.log 2023-10-21T05:35:06.241 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106304.log.gz 2023-10-21T05:35:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89565.log/var/log/ceph/ceph-client.admin.95012.log: 2023-10-21T05:35:06.241 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95012.log.gz 2023-10-21T05:35:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97311.log 2023-10-21T05:35:06.242 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98080.log.gz 2023-10-21T05:35:06.242 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88971.log 2023-10-21T05:35:06.242 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89565.log.gz 2023-10-21T05:35:06.243 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105411.log 2023-10-21T05:35:06.243 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97311.log.gz 2023-10-21T05:35:06.243 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92685.log 2023-10-21T05:35:06.243 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88971.log.gz 2023-10-21T05:35:06.244 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107750.log 2023-10-21T05:35:06.244 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105411.log.gz 2023-10-21T05:35:06.244 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103708.log 2023-10-21T05:35:06.245 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92685.log.gz 2023-10-21T05:35:06.245 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89423.log 2023-10-21T05:35:06.245 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107750.log.gz 2023-10-21T05:35:06.246 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93203.log 2023-10-21T05:35:06.246 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103708.log.gz 2023-10-21T05:35:06.246 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90215.log 2023-10-21T05:35:06.246 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89423.log.gz 2023-10-21T05:35:06.247 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105495.log 2023-10-21T05:35:06.247 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93203.log.gz 2023-10-21T05:35:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99858.log 2023-10-21T05:35:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90215.log.gz 2023-10-21T05:35:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94797.log 2023-10-21T05:35:06.248 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105495.log.gz 2023-10-21T05:35:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.99858.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.99244.log 2023-10-21T05:35:06.249 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99858.log.gz 2023-10-21T05:35:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103666.log 2023-10-21T05:35:06.250 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94797.log.gz 2023-10-21T05:35:06.250 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102768.log 2023-10-21T05:35:06.250 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99244.log.gz 2023-10-21T05:35:06.250 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99579.log 2023-10-21T05:35:06.251 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103666.log.gz 2023-10-21T05:35:06.251 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.102768.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.97906.log 2023-10-21T05:35:06.251 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102768.log.gz 2023-10-21T05:35:06.252 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.99579.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95611.log 2023-10-21T05:35:06.252 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99579.log.gz 2023-10-21T05:35:06.252 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102199.log 2023-10-21T05:35:06.252 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97906.log.gz 2023-10-21T05:35:06.253 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87498.log 2023-10-21T05:35:06.253 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95611.log.gz 2023-10-21T05:35:06.253 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89457.log 2023-10-21T05:35:06.254 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102199.log.gz 2023-10-21T05:35:06.254 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97630.log 2023-10-21T05:35:06.254 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87498.log.gz 2023-10-21T05:35:06.254 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.89457.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91393.log 2023-10-21T05:35:06.255 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89457.log.gz 2023-10-21T05:35:06.255 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90073.log 2023-10-21T05:35:06.255 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97630.log.gz 2023-10-21T05:35:06.256 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95072.log 2023-10-21T05:35:06.256 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91393.log.gz 2023-10-21T05:35:06.256 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102635.log 2023-10-21T05:35:06.256 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90073.log.gz 2023-10-21T05:35:06.257 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88664.log 2023-10-21T05:35:06.257 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95072.log.gz 2023-10-21T05:35:06.257 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98943.log 2023-10-21T05:35:06.257 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102635.log.gz 2023-10-21T05:35:06.258 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98722.log 2023-10-21T05:35:06.258 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88664.log.gz 2023-10-21T05:35:06.258 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.98943.log: --verbose -- /var/log/ceph/ceph-client.admin.87292.log 2023-10-21T05:35:06.259 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98943.log.gz 2023-10-21T05:35:06.259 INFO:teuthology.orchestra.run.smithi040.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.105369.log.gz 2023-10-21T05:35:06.259 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89332.log 2023-10-21T05:35:06.259 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98722.log.gz 2023-10-21T05:35:06.260 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96130.log 2023-10-21T05:35:06.260 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87292.log.gz 2023-10-21T05:35:06.260 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.89332.log: /var/log/ceph/ceph-client.admin.100742.log 2023-10-21T05:35:06.261 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89332.log.gz 2023-10-21T05:35:06.261 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95144.log 2023-10-21T05:35:06.261 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96130.log.gz 2023-10-21T05:35:06.261 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99903.log 2023-10-21T05:35:06.262 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100742.log.gz 2023-10-21T05:35:06.262 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96733.log 2023-10-21T05:35:06.262 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95144.log.gz 2023-10-21T05:35:06.263 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92990.log 2023-10-21T05:35:06.263 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99903.log.gz 2023-10-21T05:35:06.263 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104393.log 2023-10-21T05:35:06.263 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96733.log.gz 2023-10-21T05:35:06.264 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92745.log 2023-10-21T05:35:06.264 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92990.log.gz 2023-10-21T05:35:06.264 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89933.log 2023-10-21T05:35:06.264 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104393.log.gz 2023-10-21T05:35:06.265 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102168.log 2023-10-21T05:35:06.265 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92745.log.gz 2023-10-21T05:35:06.265 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.89933.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.107576.log 2023-10-21T05:35:06.266 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89933.log.gz 2023-10-21T05:35:06.266 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95663.log 2023-10-21T05:35:06.266 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102168.log.gz 2023-10-21T05:35:06.267 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104350.log 2023-10-21T05:35:06.267 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107576.log.gz 2023-10-21T05:35:06.267 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.95663.log: /var/log/ceph/ceph-osd.1.log 2023-10-21T05:35:06.267 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95663.log.gz 2023-10-21T05:35:06.268 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90778.log 2023-10-21T05:35:06.268 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104350.log.gz 2023-10-21T05:35:06.268 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96375.log 2023-10-21T05:35:06.269 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96977.log 2023-10-21T05:35:06.269 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90778.log.gz 2023-10-21T05:35:06.269 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88698.log 2023-10-21T05:35:06.269 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96375.log.gz 2023-10-21T05:35:06.270 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105200.log 2023-10-21T05:35:06.270 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96977.log.gz 2023-10-21T05:35:06.270 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107323.log 2023-10-21T05:35:06.270 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88698.log.gz 2023-10-21T05:35:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103840.log 2023-10-21T05:35:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105200.log.gz 2023-10-21T05:35:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97931.log 2023-10-21T05:35:06.272 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107323.log.gz 2023-10-21T05:35:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97772.log 2023-10-21T05:35:06.272 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103840.log.gz 2023-10-21T05:35:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95781.log 2023-10-21T05:35:06.273 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97931.log.gz 2023-10-21T05:35:06.273 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.97772.log: /var/log/ceph/ceph-client.admin.91886.log 2023-10-21T05:35:06.273 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97772.log.gz 2023-10-21T05:35:06.273 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.95781.log: --verbose -- /var/log/ceph/ceph-client.admin.99214.log 2023-10-21T05:35:06.274 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95781.log.gz 2023-10-21T05:35:06.274 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106472.log 2023-10-21T05:35:06.275 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91886.log.gz 2023-10-21T05:35:06.275 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90931.log 2023-10-21T05:35:06.275 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99214.log.gz 2023-10-21T05:35:06.276 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87207.log 2023-10-21T05:35:06.276 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106472.log.gz 2023-10-21T05:35:06.276 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99031.log 2023-10-21T05:35:06.276 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90931.log.gz 2023-10-21T05:35:06.277 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92775.log 2023-10-21T05:35:06.277 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87207.log.gz 2023-10-21T05:35:06.277 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99061.log 2023-10-21T05:35:06.278 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99031.log.gz 2023-10-21T05:35:06.278 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89494.log 2023-10-21T05:35:06.278 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92775.log.gz 2023-10-21T05:35:06.279 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89386.log 2023-10-21T05:35:06.279 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99061.log.gz 2023-10-21T05:35:06.279 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89224.log 2023-10-21T05:35:06.279 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89494.log.gz 2023-10-21T05:35:06.280 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105115.log 2023-10-21T05:35:06.280 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89386.log.gz 2023-10-21T05:35:06.280 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106557.log 2023-10-21T05:35:06.281 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89224.log.gz 2023-10-21T05:35:06.281 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98124.log 2023-10-21T05:35:06.281 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105115.log.gz 2023-10-21T05:35:06.281 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.106557.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.103052.log 2023-10-21T05:35:06.282 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106557.log.gz 2023-10-21T05:35:06.282 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103160.log 2023-10-21T05:35:06.283 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98124.log.gz 2023-10-21T05:35:06.283 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97341.log 2023-10-21T05:35:06.283 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103052.log.gz 2023-10-21T05:35:06.283 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90526.log 2023-10-21T05:35:06.284 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103160.log.gz 2023-10-21T05:35:06.284 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.97341.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.107023.log 2023-10-21T05:35:06.284 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97341.log.gz 2023-10-21T05:35:06.284 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91149.log 2023-10-21T05:35:06.285 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90526.log.gz 2023-10-21T05:35:06.285 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92553.log 2023-10-21T05:35:06.285 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107023.log.gz 2023-10-21T05:35:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96515.log 2023-10-21T05:35:06.286 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91149.log.gz 2023-10-21T05:35:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98782.log 2023-10-21T05:35:06.287 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92553.log.gz 2023-10-21T05:35:06.287 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98643.log 2023-10-21T05:35:06.287 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96515.log.gz 2023-10-21T05:35:06.287 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91700.log 2023-10-21T05:35:06.288 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98782.log.gz 2023-10-21T05:35:06.288 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97169.log 2023-10-21T05:35:06.288 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98643.log.gz 2023-10-21T05:35:06.289 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96575.log 2023-10-21T05:35:06.289 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91700.log.gz 2023-10-21T05:35:06.289 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94001.log 2023-10-21T05:35:06.290 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97169.log.gz 2023-10-21T05:35:06.290 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91669.log 2023-10-21T05:35:06.290 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96575.log.gz 2023-10-21T05:35:06.290 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94626.log 2023-10-21T05:35:06.291 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94001.log.gz 2023-10-21T05:35:06.291 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99955.log 2023-10-21T05:35:06.291 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91669.log.gz 2023-10-21T05:35:06.291 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103540.log 2023-10-21T05:35:06.292 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94626.log.gz 2023-10-21T05:35:06.292 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94377.log 2023-10-21T05:35:06.292 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99955.log.gz 2023-10-21T05:35:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100894.log 2023-10-21T05:35:06.293 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103540.log.gz 2023-10-21T05:35:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.94377.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.96763.log 2023-10-21T05:35:06.293 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94377.log.gz 2023-10-21T05:35:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107193.log 2023-10-21T05:35:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100894.log.gz 2023-10-21T05:35:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90614.log 2023-10-21T05:35:06.295 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96763.log.gz 2023-10-21T05:35:06.295 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.107193.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.89008.log 2023-10-21T05:35:06.295 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107193.log.gz 2023-10-21T05:35:06.295 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.90614.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.97111.log 2023-10-21T05:35:06.296 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90614.log.gz 2023-10-21T05:35:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104478.log 2023-10-21T05:35:06.296 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89008.log.gz 2023-10-21T05:35:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106937.log 2023-10-21T05:35:06.297 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97111.log.gz 2023-10-21T05:35:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89477.log 2023-10-21T05:35:06.297 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104478.log.gz 2023-10-21T05:35:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93755.log 2023-10-21T05:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106937.log.gz 2023-10-21T05:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97682.log 2023-10-21T05:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89477.log.gz 2023-10-21T05:35:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102960.log 2023-10-21T05:35:06.299 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93755.log.gz 2023-10-21T05:35:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90991.log 2023-10-21T05:35:06.299 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97682.log.gz 2023-10-21T05:35:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89116.log 2023-10-21T05:35:06.300 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102960.log.gz 2023-10-21T05:35:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101347.log 2023-10-21T05:35:06.301 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90991.log.gz 2023-10-21T05:35:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.89116.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.90570.log 2023-10-21T05:35:06.301 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89116.log.gz 2023-10-21T05:35:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99549.log 2023-10-21T05:35:06.302 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101347.log.gz 2023-10-21T05:35:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106089.log 2023-10-21T05:35:06.302 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90570.log.gz 2023-10-21T05:35:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94900.log 2023-10-21T05:35:06.303 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99549.log.gz 2023-10-21T05:35:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91544.log 2023-10-21T05:35:06.303 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106089.log.gz 2023-10-21T05:35:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.94900.log: /var/log/ceph/ceph-client.admin.94214.log 2023-10-21T05:35:06.304 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94900.log.gz 2023-10-21T05:35:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97037.log 2023-10-21T05:35:06.305 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91544.log.gz 2023-10-21T05:35:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100074.log 2023-10-21T05:35:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94214.log.gz 2023-10-21T05:35:06.306 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92715.log 2023-10-21T05:35:06.306 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97037.log.gz 2023-10-21T05:35:06.306 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99135.log 2023-10-21T05:35:06.306 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100074.log.gz 2023-10-21T05:35:06.307 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103456.log 2023-10-21T05:35:06.307 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92715.log.gz 2023-10-21T05:35:06.307 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91589.log 2023-10-21T05:35:06.308 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99135.log.gz 2023-10-21T05:35:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106768.log 2023-10-21T05:35:06.308 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103456.log.gz 2023-10-21T05:35:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94872.log 2023-10-21T05:35:06.309 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91589.log.gz 2023-10-21T05:35:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91480.log 2023-10-21T05:35:06.309 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106768.log.gz 2023-10-21T05:35:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.94872.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.88883.log 2023-10-21T05:35:06.310 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94872.log.gz 2023-10-21T05:35:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89585.log 2023-10-21T05:35:06.310 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91480.log.gz 2023-10-21T05:35:06.311 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99459.log 2023-10-21T05:35:06.311 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88883.log.gz 2023-10-21T05:35:06.311 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79805.log 2023-10-21T05:35:06.311 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89585.log.gz 2023-10-21T05:35:06.312 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91125.log 2023-10-21T05:35:06.312 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99459.log.gz 2023-10-21T05:35:06.312 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.tmp-client.admin.79805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100104.log 2023-10-21T05:35:06.312 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79805.log.gz 2023-10-21T05:35:06.313 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90349.log 2023-10-21T05:35:06.313 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91125.log.gz 2023-10-21T05:35:06.313 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87071.log 2023-10-21T05:35:06.314 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100104.log.gz 2023-10-21T05:35:06.314 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102605.log 2023-10-21T05:35:06.314 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90349.log.gz 2023-10-21T05:35:06.315 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84263.log 2023-10-21T05:35:06.315 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87071.log.gz 2023-10-21T05:35:06.315 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101803.log 2023-10-21T05:35:06.315 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102605.log.gz 2023-10-21T05:35:06.316 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.84263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100558.log 2023-10-21T05:35:06.316 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84263.log.gz 2023-10-21T05:35:06.316 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98970.log 2023-10-21T05:35:06.316 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101803.log.gz 2023-10-21T05:35:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.100558.log: -- /var/log/ceph/ceph-client.admin.100710.log 2023-10-21T05:35:06.317 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100558.log.gz 2023-10-21T05:35:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91097.log 2023-10-21T05:35:06.318 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98970.log.gz 2023-10-21T05:35:06.318 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100286.log 2023-10-21T05:35:06.318 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100710.log.gz 2023-10-21T05:35:06.318 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89261.log 2023-10-21T05:35:06.319 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91097.log.gz 2023-10-21T05:35:06.319 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87922.log 2023-10-21T05:35:06.319 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100286.log.gz 2023-10-21T05:35:06.320 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93724.log 2023-10-21T05:35:06.320 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89261.log.gz 2023-10-21T05:35:06.321 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102563.log 2023-10-21T05:35:06.321 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87922.log.gz 2023-10-21T05:35:06.321 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89511.log 2023-10-21T05:35:06.322 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93724.log.gz 2023-10-21T05:35:06.322 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.102563.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91296.log 2023-10-21T05:35:06.322 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102563.log.gz 2023-10-21T05:35:06.323 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87334.log 2023-10-21T05:35:06.323 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89511.log.gz 2023-10-21T05:35:06.323 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90822.log 2023-10-21T05:35:06.324 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91296.log.gz 2023-10-21T05:35:06.324 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89844.log 2023-10-21T05:35:06.324 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87334.log.gz 2023-10-21T05:35:06.324 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106175.log 2023-10-21T05:35:06.325 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90822.log.gz 2023-10-21T05:35:06.325 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95690.log 2023-10-21T05:35:06.325 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89844.log.gz 2023-10-21T05:35:06.326 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94735.log 2023-10-21T05:35:06.326 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106175.log.gz 2023-10-21T05:35:06.326 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95996.log 2023-10-21T05:35:06.327 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95690.log.gz 2023-10-21T05:35:06.327 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96823.log 2023-10-21T05:35:06.327 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94735.log.gz 2023-10-21T05:35:06.327 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.95996.log: --verbose -- /var/log/ceph/ceph-client.admin.102230.log 2023-10-21T05:35:06.328 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95996.log.gz 2023-10-21T05:35:06.328 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91642.log 2023-10-21T05:35:06.328 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96823.log.gz 2023-10-21T05:35:06.329 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98844.log 2023-10-21T05:35:06.329 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102230.log.gz 2023-10-21T05:35:06.329 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98395.log 2023-10-21T05:35:06.329 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91642.log.gz 2023-10-21T05:35:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89207.log 2023-10-21T05:35:06.330 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98844.log.gz 2023-10-21T05:35:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106219.log 2023-10-21T05:35:06.331 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98395.log.gz 2023-10-21T05:35:06.331 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101045.log 2023-10-21T05:35:06.331 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89207.log.gz 2023-10-21T05:35:06.331 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95567.log 2023-10-21T05:35:06.332 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106219.log.gz 2023-10-21T05:35:06.332 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89977.log 2023-10-21T05:35:06.332 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101045.log.gz 2023-10-21T05:35:06.332 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96239.log 2023-10-21T05:35:06.333 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95567.log.gz 2023-10-21T05:35:06.333 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101620.log 2023-10-21T05:35:06.333 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89977.log.gz 2023-10-21T05:35:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106979.log 2023-10-21T05:35:06.334 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96239.log.gz 2023-10-21T05:35:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87376.log 2023-10-21T05:35:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101620.log.gz 2023-10-21T05:35:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96300.log 2023-10-21T05:35:06.335 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106979.log.gz 2023-10-21T05:35:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100224.log 2023-10-21T05:35:06.335 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87376.log.gz 2023-10-21T05:35:06.336 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107618.log 2023-10-21T05:35:06.336 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96300.log.gz 2023-10-21T05:35:06.336 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.100224.log: --verbose -- /var/log/ceph/ceph-client.admin.89656.log 2023-10-21T05:35:06.337 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100224.log.gz 2023-10-21T05:35:06.337 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101317.log 2023-10-21T05:35:06.337 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107618.log.gz 2023-10-21T05:35:06.337 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93146.log 2023-10-21T05:35:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89656.log.gz 2023-10-21T05:35:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95445.log 2023-10-21T05:35:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101317.log.gz 2023-10-21T05:35:06.339 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92404.log 2023-10-21T05:35:06.339 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93146.log.gz 2023-10-21T05:35:06.339 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101106.log 2023-10-21T05:35:06.339 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95445.log.gz 2023-10-21T05:35:06.340 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94490.log 2023-10-21T05:35:06.340 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92404.log.gz 2023-10-21T05:35:06.340 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91021.log 2023-10-21T05:35:06.340 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101106.log.gz 2023-10-21T05:35:06.341 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101075.log 2023-10-21T05:35:06.341 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94490.log.gz 2023-10-21T05:35:06.341 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92352.log 2023-10-21T05:35:06.341 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91021.log.gz 2023-10-21T05:35:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95857.log 2023-10-21T05:35:06.342 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101075.log.gz 2023-10-21T05:35:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96867.log 2023-10-21T05:35:06.343 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92352.log.gz 2023-10-21T05:35:06.343 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95535.log 2023-10-21T05:35:06.343 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95857.log.gz 2023-10-21T05:35:06.343 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93050.log/var/log/ceph/ceph-client.admin.96867.log: 2023-10-21T05:35:06.344 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96867.log.gz 2023-10-21T05:35:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94305.log 2023-10-21T05:35:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95535.log.gz 2023-10-21T05:35:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105874.log 2023-10-21T05:35:06.345 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93050.log.gz 2023-10-21T05:35:06.345 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101772.log 2023-10-21T05:35:06.345 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94305.log.gz 2023-10-21T05:35:06.345 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92463.log 2023-10-21T05:35:06.346 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105874.log.gz 2023-10-21T05:35:06.346 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92107.log 2023-10-21T05:35:06.346 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101772.log.gz 2023-10-21T05:35:06.347 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88738.log 2023-10-21T05:35:06.347 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92463.log.gz 2023-10-21T05:35:06.347 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100256.log 2023-10-21T05:35:06.348 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92107.log.gz 2023-10-21T05:35:06.348 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103967.log 2023-10-21T05:35:06.348 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88738.log.gz 2023-10-21T05:35:06.348 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93324.log 2023-10-21T05:35:06.349 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100256.log.gz 2023-10-21T05:35:06.349 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99335.log 2023-10-21T05:35:06.349 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103967.log.gz 2023-10-21T05:35:06.349 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89079.log 2023-10-21T05:35:06.350 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93324.log.gz 2023-10-21T05:35:06.350 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92218.log 2023-10-21T05:35:06.350 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99335.log.gz 2023-10-21T05:35:06.350 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91730.log 2023-10-21T05:35:06.351 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89079.log.gz 2023-10-21T05:35:06.351 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103142.log 2023-10-21T05:35:06.351 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92218.log.gz 2023-10-21T05:35:06.352 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105958.log 2023-10-21T05:35:06.352 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91730.log.gz 2023-10-21T05:35:06.352 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95116.log 2023-10-21T05:35:06.352 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103142.log.gz 2023-10-21T05:35:06.353 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92158.log 2023-10-21T05:35:06.353 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105958.log.gz 2023-10-21T05:35:06.353 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90393.log 2023-10-21T05:35:06.354 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95116.log.gz 2023-10-21T05:35:06.354 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97141.log 2023-10-21T05:35:06.354 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92158.log.gz 2023-10-21T05:35:06.354 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99798.log 2023-10-21T05:35:06.355 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90393.log.gz 2023-10-21T05:35:06.355 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87838.log 2023-10-21T05:35:06.355 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97141.log.gz 2023-10-21T05:35:06.356 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91450.log 2023-10-21T05:35:06.356 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99798.log.gz 2023-10-21T05:35:06.356 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89602.log 2023-10-21T05:35:06.357 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87838.log.gz 2023-10-21T05:35:06.357 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87151.log 2023-10-21T05:35:06.357 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91450.log.gz 2023-10-21T05:35:06.358 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102738.log 2023-10-21T05:35:06.358 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89602.log.gz 2023-10-21T05:35:06.358 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107836.log 2023-10-21T05:35:06.358 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87151.log.gz 2023-10-21T05:35:06.359 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90171.log 2023-10-21T05:35:06.359 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102738.log.gz 2023-10-21T05:35:06.359 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84343.log 2023-10-21T05:35:06.359 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107836.log.gz 2023-10-21T05:35:06.360 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103215.log 2023-10-21T05:35:06.360 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90171.log.gz 2023-10-21T05:35:06.360 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107534.log 2023-10-21T05:35:06.361 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.84343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84343.log.gz 2023-10-21T05:35:06.361 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102665.log 2023-10-21T05:35:06.361 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103215.log.gz 2023-10-21T05:35:06.361 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104305.log 2023-10-21T05:35:06.362 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107534.log.gz 2023-10-21T05:35:06.362 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92879.log 2023-10-21T05:35:06.362 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102665.log.gz 2023-10-21T05:35:06.363 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94654.log 2023-10-21T05:35:06.363 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104305.log.gz 2023-10-21T05:35:06.363 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105622.log 2023-10-21T05:35:06.363 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92879.log.gz 2023-10-21T05:35:06.364 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94654.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-21T05:35:06.364 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94654.log.gz 2023-10-21T05:35:06.365 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107708.log 2023-10-21T05:35:06.365 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105622.log.gz 2023-10-21T05:35:06.365 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89025.log 2023-10-21T05:35:06.365 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105158.log 2023-10-21T05:35:06.366 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107708.log.gz 2023-10-21T05:35:06.366 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94705.log 2023-10-21T05:35:06.366 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89025.log.gz 2023-10-21T05:35:06.367 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105916.log 2023-10-21T05:35:06.367 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105158.log.gz 2023-10-21T05:35:06.367 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97834.log 2023-10-21T05:35:06.367 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94705.log.gz 2023-10-21T05:35:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99163.log 2023-10-21T05:35:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105916.log.gz 2023-10-21T05:35:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90122.log 2023-10-21T05:35:06.368 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97834.log.gz 2023-10-21T05:35:06.369 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94678.log 2023-10-21T05:35:06.369 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99163.log.gz 2023-10-21T05:35:06.369 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101196.log 2023-10-21T05:35:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90122.log.gz 2023-10-21T05:35:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101015.log 2023-10-21T05:35:06.370 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94678.log.gz 2023-10-21T05:35:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101925.log 2023-10-21T05:35:06.371 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101196.log.gz 2023-10-21T05:35:06.371 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.101015.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.101986.log 2023-10-21T05:35:06.371 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101015.log.gz 2023-10-21T05:35:06.372 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87880.log 2023-10-21T05:35:06.372 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101925.log.gz 2023-10-21T05:35:06.372 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101650.log 2023-10-21T05:35:06.372 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101986.log.gz 2023-10-21T05:35:06.373 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99828.log 2023-10-21T05:35:06.373 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87880.log.gz 2023-10-21T05:35:06.373 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98695.log 2023-10-21T05:35:06.373 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101650.log.gz 2023-10-21T05:35:06.374 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99432.log 2023-10-21T05:35:06.374 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99828.log.gz 2023-10-21T05:35:06.374 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103751.log 2023-10-21T05:35:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98695.log.gz 2023-10-21T05:35:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91206.log 2023-10-21T05:35:06.375 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99432.log.gz 2023-10-21T05:35:06.376 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.103751.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.90874.log 2023-10-21T05:35:06.376 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103751.log.gz 2023-10-21T05:35:06.376 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99519.log 2023-10-21T05:35:06.377 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91206.log.gz 2023-10-21T05:35:06.377 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99187.log 2023-10-21T05:35:06.377 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90874.log.gz 2023-10-21T05:35:06.378 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.99519.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.98419.log 2023-10-21T05:35:06.378 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99519.log.gz 2023-10-21T05:35:06.378 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100194.log 2023-10-21T05:35:06.379 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99187.log.gz 2023-10-21T05:35:06.379 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89170.log 2023-10-21T05:35:06.379 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98419.log.gz 2023-10-21T05:35:06.379 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98891.log 2023-10-21T05:35:06.380 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100194.log.gz 2023-10-21T05:35:06.380 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87750.log 2023-10-21T05:35:06.380 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89170.log.gz 2023-10-21T05:35:06.381 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92658.log 2023-10-21T05:35:06.381 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98891.log.gz 2023-10-21T05:35:06.381 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102380.log 2023-10-21T05:35:06.382 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87750.log.gz 2023-10-21T05:35:06.382 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90481.log 2023-10-21T05:35:06.382 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92658.log.gz 2023-10-21T05:35:06.382 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97958.log 2023-10-21T05:35:06.383 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102380.log.gz 2023-10-21T05:35:06.383 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107878.log 2023-10-21T05:35:06.383 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90481.log.gz 2023-10-21T05:35:06.384 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.97958.log: -- /var/log/ceph/ceph-client.admin.97250.log 2023-10-21T05:35:06.384 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97958.log.gz 2023-10-21T05:35:06.384 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91236.log 2023-10-21T05:35:06.384 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107878.log.gz 2023-10-21T05:35:06.385 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87752.log 2023-10-21T05:35:06.385 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97250.log.gz 2023-10-21T05:35:06.385 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88846.log 2023-10-21T05:35:06.386 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91236.log.gz 2023-10-21T05:35:06.386 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93694.log 2023-10-21T05:35:06.386 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87752.log.gz 2023-10-21T05:35:06.387 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88846.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-21T05:35:06.387 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88846.log.gz 2023-10-21T05:35:06.387 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.93694.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.107922.log 2023-10-21T05:35:06.387 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93694.log.gz 2023-10-21T05:35:06.388 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102320.log 2023-10-21T05:35:06.388 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91834.log 2023-10-21T05:35:06.388 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107922.log.gz 2023-10-21T05:35:06.388 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92634.log 2023-10-21T05:35:06.389 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102320.log.gz 2023-10-21T05:35:06.389 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100346.log 2023-10-21T05:35:06.389 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91834.log.gz 2023-10-21T05:35:06.390 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106388.log 2023-10-21T05:35:06.390 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92634.log.gz 2023-10-21T05:35:06.390 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.100346.log: --verbose -- /var/log/ceph/ceph-client.admin.106002.log 2023-10-21T05:35:06.390 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100346.log.gz 2023-10-21T05:35:06.391 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106852.log 2023-10-21T05:35:06.391 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106388.log.gz 2023-10-21T05:35:06.391 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88829.log 2023-10-21T05:35:06.392 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106002.log.gz 2023-10-21T05:35:06.392 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94767.log 2023-10-21T05:35:06.392 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106852.log.gz 2023-10-21T05:35:06.392 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102941.log 2023-10-21T05:35:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88829.log.gz 2023-10-21T05:35:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102290.log 2023-10-21T05:35:06.393 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94767.log.gz 2023-10-21T05:35:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96485.log 2023-10-21T05:35:06.394 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102941.log.gz 2023-10-21T05:35:06.394 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87624.log 2023-10-21T05:35:06.394 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102290.log.gz 2023-10-21T05:35:06.395 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91944.log 2023-10-21T05:35:06.395 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96485.log.gz 2023-10-21T05:35:06.395 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105537.log 2023-10-21T05:35:06.395 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87624.log.gz 2023-10-21T05:35:06.396 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97658.log 2023-10-21T05:35:06.396 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91944.log.gz 2023-10-21T05:35:06.396 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89062.log 2023-10-21T05:35:06.397 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105537.log.gz 2023-10-21T05:35:06.397 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103624.log 2023-10-21T05:35:06.397 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97658.log.gz 2023-10-21T05:35:06.397 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95225.log 2023-10-21T05:35:06.398 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89062.log.gz 2023-10-21T05:35:06.398 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.103624.log: --verbose -- /var/log/ceph/ceph-client.admin.105453.log 2023-10-21T05:35:06.398 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103624.log.gz 2023-10-21T05:35:06.399 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94924.log 2023-10-21T05:35:06.399 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95225.log.gz 2023-10-21T05:35:06.399 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95966.log 2023-10-21T05:35:06.399 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105453.log.gz 2023-10-21T05:35:06.400 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95720.log 2023-10-21T05:35:06.400 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94924.log.gz 2023-10-21T05:35:06.400 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95394.log 2023-10-21T05:35:06.401 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95966.log.gz 2023-10-21T05:35:06.401 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92806.log 2023-10-21T05:35:06.401 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95720.log.gz 2023-10-21T05:35:06.401 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95394.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-21T05:35:06.402 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95394.log.gz 2023-10-21T05:35:06.402 INFO:teuthology.orchestra.run.smithi040.stderr: 90.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-21T05:35:06.402 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104094.log 2023-10-21T05:35:06.402 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92806.log.gz 2023-10-21T05:35:06.403 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90030.log 2023-10-21T05:35:06.403 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94951.log 2023-10-21T05:35:06.403 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104094.log.gz 2023-10-21T05:35:06.403 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96793.log 2023-10-21T05:35:06.404 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90030.log.gz 2023-10-21T05:35:06.404 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92380.log 2023-10-21T05:35:06.404 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94951.log.gz 2023-10-21T05:35:06.404 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92131.log 2023-10-21T05:35:06.405 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96793.log.gz 2023-10-21T05:35:06.405 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89710.log/var/log/ceph/ceph-client.admin.92380.log: 2023-10-21T05:35:06.405 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92380.log.gz 2023-10-21T05:35:06.406 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92131.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-21T05:35:06.406 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92131.log.gz 2023-10-21T05:35:06.406 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94061.log 2023-10-21T05:35:06.406 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89710.log.gz 2023-10-21T05:35:06.407 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100955.log/var/log/ceph/ceph-osd.2.log: 2023-10-21T05:35:06.407 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.94061.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.88647.log 2023-10-21T05:35:06.407 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94061.log.gz 2023-10-21T05:35:06.408 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89887.log 2023-10-21T05:35:06.408 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100955.log.gz 2023-10-21T05:35:06.408 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.88647.log: /var/log/ceph/ceph-client.admin.107661.log 2023-10-21T05:35:06.409 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88647.log.gz 2023-10-21T05:35:06.409 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90305.log 2023-10-21T05:35:06.409 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89887.log.gz 2023-10-21T05:35:06.409 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94031.log 2023-10-21T05:35:06.410 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107661.log.gz 2023-10-21T05:35:06.410 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96620.log 2023-10-21T05:35:06.410 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90305.log.gz 2023-10-21T05:35:06.411 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94458.log 2023-10-21T05:35:06.411 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94031.log.gz 2023-10-21T05:35:06.411 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96620.log.gzgzip 2023-10-21T05:35:06.411 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.99380.log 2023-10-21T05:35:06.412 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94458.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.94458.log.gz 2023-10-21T05:35:06.412 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.89619.log 2023-10-21T05:35:06.412 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91369.log 2023-10-21T05:35:06.412 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99380.log.gz 2023-10-21T05:35:06.413 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89619.log.gz 2023-10-21T05:35:06.413 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93667.log 2023-10-21T05:35:06.413 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91369.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.108136.log 2023-10-21T05:35:06.413 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91369.log.gz 2023-10-21T05:35:06.414 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93667.log.gz 2023-10-21T05:35:06.414 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107492.log 2023-10-21T05:35:06.414 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.108136.log: -- /var/log/ceph/ceph-client.admin.106894.log 2023-10-21T05:35:06.414 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108136.log.gz 2023-10-21T05:35:06.415 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107492.log.gz 2023-10-21T05:35:06.415 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95366.log 2023-10-21T05:35:06.415 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101834.log 2023-10-21T05:35:06.416 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106894.log.gz 2023-10-21T05:35:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96701.log 2023-10-21T05:35:06.416 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95366.log.gz 2023-10-21T05:35:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106600.log 2023-10-21T05:35:06.417 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101834.log.gz 2023-10-21T05:35:06.417 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96701.log: 89.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-21T05:35:06.417 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96701.log.gz 2023-10-21T05:35:06.417 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102107.log 2023-10-21T05:35:06.418 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92903.log 2023-10-21T05:35:06.418 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106600.log.gz 2023-10-21T05:35:06.418 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102796.log 2023-10-21T05:35:06.419 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102107.log.gz 2023-10-21T05:35:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93122.log 2023-10-21T05:35:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92903.log.gz 2023-10-21T05:35:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98293.log 2023-10-21T05:35:06.420 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102796.log.gz 2023-10-21T05:35:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90748.log 2023-10-21T05:35:06.420 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93122.log.gz 2023-10-21T05:35:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102869.log 2023-10-21T05:35:06.421 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98293.log.gz 2023-10-21T05:35:06.421 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103414.log 2023-10-21T05:35:06.421 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90748.log.gz 2023-10-21T05:35:06.422 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93397.log 2023-10-21T05:35:06.422 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102869.log.gz 2023-10-21T05:35:06.422 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.103414.log: /var/log/ceph/ceph-client.admin.105580.log 2023-10-21T05:35:06.422 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103414.log.gz 2023-10-21T05:35:06.423 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100467.log 2023-10-21T05:35:06.423 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93397.log.gz 2023-10-21T05:35:06.423 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97556.log 2023-10-21T05:35:06.423 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105580.log.gz 2023-10-21T05:35:06.424 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95751.log 2023-10-21T05:35:06.424 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100467.log.gz 2023-10-21T05:35:06.424 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94595.log 2023-10-21T05:35:06.425 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97556.log.gz 2023-10-21T05:35:06.425 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93888.log 2023-10-21T05:35:06.425 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95751.log.gz 2023-10-21T05:35:06.425 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94827.log 2023-10-21T05:35:06.426 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94595.log.gz 2023-10-21T05:35:06.426 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95418.log 2023-10-21T05:35:06.426 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93888.log.gz 2023-10-21T05:35:06.426 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.94827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84105.log 2023-10-21T05:35:06.427 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94827.log.gz 2023-10-21T05:35:06.427 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89153.log 2023-10-21T05:35:06.427 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95418.log.gz 2023-10-21T05:35:06.428 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.84105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99275.log 2023-10-21T05:35:06.428 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84105.log.gz 2023-10-21T05:35:06.428 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104947.log 2023-10-21T05:35:06.428 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89153.log.gz 2023-10-21T05:35:06.429 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98476.log 2023-10-21T05:35:06.429 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99275.log.gz 2023-10-21T05:35:06.429 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.104947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97414.log 2023-10-21T05:35:06.430 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104947.log.gz 2023-10-21T05:35:06.430 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98568.log 2023-10-21T05:35:06.430 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98476.log.gz 2023-10-21T05:35:06.430 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87418.log 2023-10-21T05:35:06.431 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97414.log.gz 2023-10-21T05:35:06.431 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.98568.log: /var/log/ceph/ceph-client.admin.103088.log 2023-10-21T05:35:06.431 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98568.log.gz 2023-10-21T05:35:06.431 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93421.log 2023-10-21T05:35:06.432 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87418.log.gz 2023-10-21T05:35:06.432 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95286.log 2023-10-21T05:35:06.432 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103088.log.gz 2023-10-21T05:35:06.433 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105790.log 2023-10-21T05:35:06.433 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93421.log.gz 2023-10-21T05:35:06.433 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.95286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93971.log 2023-10-21T05:35:06.433 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95286.log.gz 2023-10-21T05:35:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91790.log 2023-10-21T05:35:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105790.log.gz 2023-10-21T05:35:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96455.log/var/log/ceph/ceph-client.admin.93971.log: 2023-10-21T05:35:06.435 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93971.log.gz 2023-10-21T05:35:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103070.log 2023-10-21T05:35:06.435 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91790.log.gz 2023-10-21T05:35:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92930.log 2023-10-21T05:35:06.436 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96455.log.gz 2023-10-21T05:35:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91617.log 2023-10-21T05:35:06.436 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103070.log.gz 2023-10-21T05:35:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.92930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88991.log 2023-10-21T05:35:06.437 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92930.log.gz 2023-10-21T05:35:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99737.log 2023-10-21T05:35:06.437 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91617.log.gz 2023-10-21T05:35:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101257.log 2023-10-21T05:35:06.438 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88991.log.gz 2023-10-21T05:35:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.99737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88681.log 2023-10-21T05:35:06.438 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99737.log.gz 2023-10-21T05:35:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105664.log 2023-10-21T05:35:06.439 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101257.log.gz 2023-10-21T05:35:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100497.log 2023-10-21T05:35:06.440 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.88681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88681.log.gz 2023-10-21T05:35:06.440 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.105664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103882.log 2023-10-21T05:35:06.440 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105664.log.gz 2023-10-21T05:35:06.440 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.100497.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.93643.log 2023-10-21T05:35:06.441 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100497.log.gz 2023-10-21T05:35:06.441 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107280.log 2023-10-21T05:35:06.441 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103882.log.gz 2023-10-21T05:35:06.442 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.93643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91176.log 2023-10-21T05:35:06.442 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93643.log.gz 2023-10-21T05:35:06.442 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.107280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101498.log 2023-10-21T05:35:06.442 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107280.log.gz 2023-10-21T05:35:06.443 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96403.log 2023-10-21T05:35:06.443 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91176.log.gz 2023-10-21T05:35:06.443 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.101498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103288.log 2023-10-21T05:35:06.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90697.log 2023-10-21T05:35:06.444 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96403.log.gz 2023-10-21T05:35:06.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97526.log 2023-10-21T05:35:06.444 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103288.log.gz 2023-10-21T05:35:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.90697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103582.log 2023-10-21T05:35:06.445 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90697.log.gz 2023-10-21T05:35:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.97526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87625.log 2023-10-21T05:35:06.446 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97526.log.gz 2023-10-21T05:35:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102923.log 2023-10-21T05:35:06.446 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103582.log.gz 2023-10-21T05:35:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98176.log 2023-10-21T05:35:06.447 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87625.log.gz 2023-10-21T05:35:06.447 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102443.log 2023-10-21T05:35:06.447 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102923.log.gz 2023-10-21T05:35:06.447 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96919.log 2023-10-21T05:35:06.448 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98176.log.gz 2023-10-21T05:35:06.448 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102350.log 2023-10-21T05:35:06.448 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102443.log.gz 2023-10-21T05:35:06.449 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.96919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102047.log 2023-10-21T05:35:06.449 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96919.log.gz 2023-10-21T05:35:06.449 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102350.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102350.log.gz 2023-10-21T05:35:06.449 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91051.log 2023-10-21T05:35:06.450 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.102047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91513.log 2023-10-21T05:35:06.450 INFO:teuthology.orchestra.run.smithi040.stderr: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.101498.log.gz 2023-10-21T05:35:06.450 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102047.log.gz 2023-10-21T05:35:06.451 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87499.log 2023-10-21T05:35:06.451 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91051.log.gz 2023-10-21T05:35:06.451 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98919.log 2023-10-21T05:35:06.451 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91513.log.gz 2023-10-21T05:35:06.452 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.87499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103330.log 2023-10-21T05:35:06.452 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87499.log.gz 2023-10-21T05:35:06.452 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.98919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100527.log 2023-10-21T05:35:06.453 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98919.log.gz 2023-10-21T05:35:06.453 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.103330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106262.log 2023-10-21T05:35:06.453 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103330.log.gz 2023-10-21T05:35:06.453 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.100527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102016.log 2023-10-21T05:35:06.454 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100527.log.gz 2023-10-21T05:35:06.454 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91760.log 2023-10-21T05:35:06.454 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106262.log.gz 2023-10-21T05:35:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.102016.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.89349.log 2023-10-21T05:35:06.455 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102016.log.gz 2023-10-21T05:35:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.91760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106047.log 2023-10-21T05:35:06.455 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91760.log.gz 2023-10-21T05:35:06.456 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.89349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89349.log.gz 2023-10-21T05:35:06.456 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.106047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106047.log.gz 2023-10-21T05:35:33.017 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-21T05:35:43.716 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-21T06:20:00.907 INFO:teuthology.orchestra.run.smithi040.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-21T06:33:50.938 INFO:teuthology.orchestra.run.smithi040.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-21T06:33:58.869 INFO:teuthology.orchestra.run.smithi040.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-21T06:33:58.871 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-21T06:33:58.872 INFO:teuthology.orchestra.run.smithi040.stderr:real 58m53.109s 2023-10-21T06:33:58.872 INFO:teuthology.orchestra.run.smithi040.stderr:user 20m48.547s 2023-10-21T06:33:58.872 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m59.235s 2023-10-21T06:33:58.873 INFO:tasks.ceph:Archiving logs... 2023-10-21T06:33:58.874 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357/remote/smithi040/log 2023-10-21T06:33:58.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-21T06:36:44.311 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T06:36:44.337 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-21T06:36:44.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-21T06:36:44.398 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-21T06:36:44.407 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-21T06:36:44.408 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T06:36:44.424 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2023-10-21T06:36:44.510 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-21T06:36:44.510 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-10-21T06:36:44.511 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +783us[ +783us] +/- 87ms 2023-10-21T06:36:44.511 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -1411us[-1435us] +/- 53ms 2023-10-21T06:36:44.511 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +1123us[+1123us] +/- 66ms 2023-10-21T06:36:44.512 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-21T06:36:44.513 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-21T06:36:44.522 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-21T06:36:44.523 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-21T06:36:44.532 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-21T06:36:44.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-21T06:36:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:36:44.802 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-21T06:36:44.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-21T06:36:44.873 INFO:teuthology.task.internal:Duration was 9915.003271 seconds 2023-10-21T06:36:44.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-21T06:36:44.883 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-21T06:36:44.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-21T06:36:44.985 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-21T06:36:45.411 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-21T06:36:45.411 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-10-21T06:36:45.412 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-21T06:36:45.438 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-21T06:36:45.439 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-21T06:36:45.522 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-21T06:36:45.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-21T06:36:45.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-21T06:36:45.884 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-21T06:36:45.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-21T06:36:45.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-21T06:36:45.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:36:46.005 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-10-21T06:36:46.040 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:36:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:36:46.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-21T06:36:46.066 INFO:teuthology.task.internal:Transferring archived files... 2023-10-21T06:36:46.068 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357/remote/smithi040 2023-10-21T06:36:46.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-21T06:36:46.138 INFO:teuthology.task.internal:Removing archive directory... 2023-10-21T06:36:46.138 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-21T06:36:46.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-21T06:36:46.198 INFO:teuthology.task.internal:Not uploading archives. 2023-10-21T06:36:46.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-21T06:36:46.208 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-21T06:36:46.209 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-21T06:36:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 21 06:36 /home/ubuntu/cephtest 2023-10-21T06:36:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: 403116 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 21 04:00 /home/ubuntu/cephtest/mnt.0 2023-10-21T06:36:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: 403117 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 21 04:01 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-21T06:36:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: 417020 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 21 04:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-21T06:36:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:36:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-21T06:36:46.266 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:36:46.276 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-21T06:36:46.287 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-21T06:36:46.299 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:36:46.300 INFO:teuthology.nuke:Checking targets against current locks 2023-10-21T06:36:46.338 DEBUG:teuthology.nuke:shortname: smithi040 2023-10-21T06:36:46.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T06:36:46.365 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:42:57.197671', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAfZOxwwOSOg7SAAW1HKu994xk9gt1F1rsH50yetsOqqRSjPNPyrelDEyxFF3hzsEyq0o6huOWqCT9bTgCmORpc='} 2023-10-21T06:36:46.414 INFO:teuthology.orchestra.console.smithi040:Power off 2023-10-21T06:36:46.415 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T06:36:46.438 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2023-10-21T06:36:46.439 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:36:46.462 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-10-21T06:36:50.464 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:36:50.573 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-10-21T06:36:54.575 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:36:54.685 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-10-21T06:36:58.686 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:36:58.797 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2023-10-21T06:36:58.898 INFO:teuthology.orchestra.console.smithi040:Power off completed 2023-10-21T06:36:59.004 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} duration: 9915.003271102905 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi040 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c7a987d078454d378ec209776a2fd3cf status: fail success: false 2023-10-21T06:36:59.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T06:36:59.077 INFO:teuthology.run:FAIL