2023-09-18T20:25:53.850 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-18T20:25:53.851 DEBUG:teuthology.run:Teuthology command: teuthology --name rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi --description rbd:cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} --owner scheduled_rraja@teuthology --verbose --archive /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734 -- /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734/orig.config.yaml 2023-09-18T20:25:53.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-18T20:25:53.943 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734 branch: wip-ajarr-repeated-rbd-client-blocklisting description: rbd:cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7397734' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-ajarr-repeated-rbd-client-blocklisting ceph: conf: client: rbd default features: 1 global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug_rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 277a824d757a39499f35213b338ab8a8372df0f6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 277a824d757a39499f35213b338ab8a8372df0f6 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ajarr-repeated-rbd-client-blocklisting-2 sha1: daad63c9e55979797c86b6218776b5cf580dafc3 owner: scheduled_rraja@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1851 sha1: 277a824d757a39499f35213b338ab8a8372df0f6 sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-ajarr-repeated-rbd-client-blocklisting-2 suite_path: /home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa suite_relpath: qa suite_repo: https://github.com/ajarr/ceph suite_sha1: daad63c9e55979797c86b6218776b5cf580dafc3 targets: smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: null - ceph: null - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-18_20:19:35 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1394 2023-09-18T20:25:53.944 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa; will attempt to use it 2023-09-18T20:25:53.944 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa/tasks 2023-09-18T20:25:53.945 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-18T20:25:53.946 INFO:teuthology.task.internal:Checking packages... 2023-09-18T20:25:53.968 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '277a824d757a39499f35213b338ab8a8372df0f6' 2023-09-18T20:25:53.969 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-18T20:25:53.969 INFO:teuthology.packaging:ref: None 2023-09-18T20:25:53.969 INFO:teuthology.packaging:tag: None 2023-09-18T20:25:53.970 INFO:teuthology.packaging:branch: wip-ajarr-repeated-rbd-client-blocklisting 2023-09-18T20:25:53.970 INFO:teuthology.packaging:sha1: 277a824d757a39499f35213b338ab8a8372df0f6 2023-09-18T20:25:53.970 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-ajarr-repeated-rbd-client-blocklisting 2023-09-18T20:25:54.122 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6170.g277a824d 2023-09-18T20:25:54.122 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-18T20:25:54.134 INFO:teuthology.task.internal:no buildpackages task found 2023-09-18T20:25:54.134 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-18T20:25:54.162 INFO:teuthology.task.internal:Saving configuration 2023-09-18T20:25:54.173 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-18T20:25:54.189 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-18T20:25:54.212 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-18 20:20:25.695614', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-18T20:25:54.212 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-18T20:25:54.219 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-18T20:25:54.219 INFO:teuthology.run_tasks:Running task console_log... 2023-09-18T20:25:54.266 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f87a83c7940>, signals=[15]) 2023-09-18T20:25:54.269 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-18T20:25:54.280 INFO:teuthology.task.internal:Opening connections... 2023-09-18T20:25:54.281 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:25:54.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-18T20:25:54.349 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-18T20:25:54.358 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2023-09-18T20:25:54.377 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2023-09-18T20:25:54.378 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2023-09-18T20:25:54.436 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Red Hat Enterprise Linux" 2023-09-18T20:25:54.436 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8.6 (Ootpa)" 2023-09-18T20:25:54.437 INFO:teuthology.orchestra.run.smithi149.stdout:ID="rhel" 2023-09-18T20:25:54.437 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="fedora" 2023-09-18T20:25:54.437 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8.6" 2023-09-18T20:25:54.437 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2023-09-18T20:25:54.437 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-09-18T20:25:54.438 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2023-09-18T20:25:54.438 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-09-18T20:25:54.438 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.redhat.com/" 2023-09-18T20:25:54.438 INFO:teuthology.orchestra.run.smithi149.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-18T20:25:54.438 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-18T20:25:54.439 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:25:54.439 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-18T20:25:54.439 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-09-18T20:25:54.439 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-18T20:25:54.439 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-09-18T20:25:54.440 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2023-09-18T20:25:54.465 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-18T20:25:54.474 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-18T20:25:54.485 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-18T20:25:54.485 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2023-09-18T20:25:54.504 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-18T20:25:54.515 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-18T20:25:54.515 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2023-09-18T20:25:54.569 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-18T20:25:54.611 INFO:teuthology.run_tasks:Running task kernel... 2023-09-18T20:25:54.650 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-18T20:25:54.651 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-18T20:25:54.651 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-18T20:25:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-18T20:25:54.670 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2023-09-18T20:25:54.729 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-372.9.1.el8.x86_64 2023-09-18T20:25:54.729 INFO:teuthology.task.kernel:Running kernel on smithi149: 4.18.0-372.9.1.el8.x86_64 2023-09-18T20:25:54.730 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2023-09-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2023-09-18T20:25:57.507 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 9.4 kB/s | 3.5 kB 00:00 2023-09-18T20:25:58.416 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-09-18T20:26:01.891 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 363 kB/s | 24 kB 00:00 2023-09-18T20:26:02.602 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-09-18T20:26:02.652 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:26:02.653 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:26:02.653 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:26:02.785 DEBUG:teuthology.orchestra.run.smithi149:> echo no | sudo yum reinstall kernel || true 2023-09-18T20:26:03.116 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:26:03.117 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2023-09-18T20:26:03.383 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:02 ago on Mon 18 Sep 2023 08:26:01 PM UTC. 2023-09-18T20:26:03.425 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-09-18T20:26:03.480 INFO:teuthology.orchestra.run.smithi149.stderr:Error: No packages marked for reinstall. 2023-09-18T20:26:03.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum reinstall -y kernel || true 2023-09-18T20:26:03.858 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:26:03.859 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2023-09-18T20:26:04.119 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:03 ago on Mon 18 Sep 2023 08:26:01 PM UTC. 2023-09-18T20:26:04.161 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-09-18T20:26:04.203 INFO:teuthology.orchestra.run.smithi149.stderr:Error: No packages marked for reinstall. 2023-09-18T20:26:04.250 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2023-09-18T20:26:04.332 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-09-18T20:26:04.333 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-09-18T20:26:04.333 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-09-18T20:26:04.333 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-18T20:26:04.334 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-18T20:26:04.363 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2023-09-18T20:26:04.380 DEBUG:teuthology.parallel:result is None 2023-09-18T20:26:04.380 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-18T20:26:04.390 INFO:teuthology.task.internal:Creating test directory... 2023-09-18T20:26:04.391 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-18T20:26:04.436 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-18T20:26:04.448 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-18T20:26:04.469 INFO:teuthology.task.internal:Creating archive directory... 2023-09-18T20:26:04.469 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-18T20:26:04.498 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-18T20:26:04.507 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-18T20:26:04.507 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-18T20:26:04.578 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-18T20:26:04.588 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-18T20:26:04.590 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-18T20:26:04.599 INFO:teuthology.task.internal:Configuring sudo... 2023-09-18T20:26:04.600 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-18T20:26:04.626 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-18T20:26:04.640 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-18T20:26:04.640 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-18T20:26:04.682 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-18T20:26:04.766 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-18T20:26:04.814 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-18T20:26:04.901 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-18T20:26:04.941 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:26:04.941 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-18T20:26:05.018 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2023-09-18T20:26:05.114 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-18T20:26:05.407 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-18T20:26:05.416 INFO:teuthology.task.internal:Starting timer... 2023-09-18T20:26:05.417 INFO:teuthology.run_tasks:Running task pcp... 2023-09-18T20:26:05.452 INFO:teuthology.run_tasks:Running task selinux... 2023-09-18T20:26:05.465 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-18T20:26:05.466 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2023-09-18T20:26:05.535 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2023-09-18T20:26:05.537 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-18T20:26:05.537 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2023-09-18T20:26:05.557 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2023-09-18T20:26:05.557 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi149.front.sepia.ceph.com': 'permissive'} 2023-09-18T20:26:05.557 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-18T20:26:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-18T20:26:05.625 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-18T20:26:05.625 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2023-09-18T20:26:05.693 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-18T20:26:05.702 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-18T20:26:05.708 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-18T20:26:05.876 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-18T20:26:05.892 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-18T20:26:05.893 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-18T20:34:32.210 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2023-09-18T20:34:32.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-09-18T20:34:32.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-18T20:34:32.283 DEBUG:teuthology.orchestra.run.smithi149:> true 2023-09-18T20:34:32.309 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-09-18T20:34:32.309 INFO:teuthology.run_tasks:Running task clock... 2023-09-18T20:34:32.319 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-18T20:34:32.319 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-18T20:34:32.320 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-18T20:34:32.382 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-18T20:34:32.397 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-18T20:34:32.467 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2023-09-18T20:34:32.479 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2023-09-18T20:34:32.494 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-18T20:34:32.508 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-18T20:34:32.588 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2023-09-18T20:34:32.592 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-18T20:34:32.592 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2023-09-18T20:34:32.592 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-18T20:34:32.592 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-18T20:34:32.593 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-18T20:34:32.593 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-18T20:34:32.594 INFO:teuthology.run_tasks:Running task install... 2023-09-18T20:34:32.602 DEBUG:teuthology.task.install:project ceph 2023-09-18T20:34:32.602 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6'}} 2023-09-18T20:34:32.602 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6'} 2023-09-18T20:34:32.603 INFO:teuthology.task.install:Using flavor: default 2023-09-18T20:34:32.608 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-18T20:34:32.608 INFO:teuthology.task.install:extra packages: [] 2023-09-18T20:34:32.609 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6', 'tag': None, 'wait_for_package': False} 2023-09-18T20:34:32.610 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=277a824d757a39499f35213b338ab8a8372df0f6 2023-09-18T20:34:32.773 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/ 2023-09-18T20:34:32.773 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6170.g277a824d 2023-09-18T20:34:32.952 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-18T20:34:32.953 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:34:32.953 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-18T20:34:32.980 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-18T20:34:32.981 DEBUG:teuthology.orchestra.run.smithi149:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/277a824d757a39499f35213b338ab8a8372df0f6/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-18T20:34:33.049 DEBUG:teuthology.orchestra.run.smithi149:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-18T20:34:33.127 DEBUG:teuthology.orchestra.run.smithi149:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-18T20:34:33.193 INFO:teuthology.orchestra.run.smithi149.stdout:check_obsoletes = 1 2023-09-18T20:34:33.195 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum clean all 2023-09-18T20:34:33.571 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:34:33.863 INFO:teuthology.orchestra.run.smithi149.stdout:44 files removed 2023-09-18T20:34:33.911 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-radosgw 2023-09-18T20:34:34.242 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:34:35.129 INFO:teuthology.orchestra.run.smithi149.stdout:ceph packages for x86_64 165 kB/s | 81 kB 00:00 2023-09-18T20:34:35.480 INFO:teuthology.orchestra.run.smithi149.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-09-18T20:34:35.805 INFO:teuthology.orchestra.run.smithi149.stdout:ceph source packages 4.9 kB/s | 1.5 kB 00:00 2023-09-18T20:34:35.956 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-09-18T20:34:37.482 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-09-18T20:34:41.024 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 361 kB/s | 24 kB 00:00 2023-09-18T20:34:41.950 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-09-18T20:34:50.043 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-09-18T20:35:01.213 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:35:01.215 INFO:teuthology.orchestra.run.smithi149.stdout:==================================================================================================== 2023-09-18T20:35:01.216 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:35:01.216 INFO:teuthology.orchestra.run.smithi149.stdout:==================================================================================================== 2023-09-18T20:35:01.216 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:35:01.216 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw x86_64 2:18.0.0-6170.g277a824d.el8 ceph 14 M 2023-09-18T20:35:01.216 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrading: 2023-09-18T20:35:01.217 INFO:teuthology.orchestra.run.smithi149.stdout: librados2 x86_64 2:18.0.0-6170.g277a824d.el8 ceph 3.7 M 2023-09-18T20:35:01.217 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1 x86_64 2:18.0.0-6170.g277a824d.el8 ceph 3.7 M 2023-09-18T20:35:01.217 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:35:01.217 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base x86_64 2:18.0.0-6170.g277a824d.el8 ceph 5.4 M 2023-09-18T20:35:01.217 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common x86_64 2:18.0.0-6170.g277a824d.el8 ceph 23 M 2023-09-18T20:35:01.218 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux x86_64 2:18.0.0-6170.g277a824d.el8 ceph 24 k 2023-09-18T20:35:01.218 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-18T20:35:01.218 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2 x86_64 2:18.0.0-6170.g277a824d.el8 ceph 827 k 2023-09-18T20:35:01.218 INFO:teuthology.orchestra.run.smithi149.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-18T20:35:01.218 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-18T20:35:01.219 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1 x86_64 2:18.0.0-6170.g277a824d.el8 ceph 518 k 2023-09-18T20:35:01.219 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-09-18T20:35:01.219 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2 x86_64 2:18.0.0-6170.g277a824d.el8 ceph 6.9 M 2023-09-18T20:35:01.219 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-18T20:35:01.219 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-18T20:35:01.220 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse x86_64 2:18.0.0-6170.g277a824d.el8 ceph 45 k 2023-09-18T20:35:01.220 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common x86_64 2:18.0.0-6170.g277a824d.el8 ceph 126 k 2023-09-18T20:35:01.220 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs x86_64 2:18.0.0-6170.g277a824d.el8 ceph 196 k 2023-09-18T20:35:01.220 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-18T20:35:01.220 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados x86_64 2:18.0.0-6170.g277a824d.el8 ceph 371 k 2023-09-18T20:35:01.220 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd x86_64 2:18.0.0-6170.g277a824d.el8 ceph 364 k 2023-09-18T20:35:01.221 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw x86_64 2:18.0.0-6170.g277a824d.el8 ceph 107 k 2023-09-18T20:35:01.221 INFO:teuthology.orchestra.run.smithi149.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-18T20:35:01.221 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:01.221 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:35:01.221 INFO:teuthology.orchestra.run.smithi149.stdout:==================================================================================================== 2023-09-18T20:35:01.221 INFO:teuthology.orchestra.run.smithi149.stdout:Install 21 Packages 2023-09-18T20:35:01.222 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrade 2 Packages 2023-09-18T20:35:01.222 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:01.222 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 62 M 2023-09-18T20:35:01.223 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:35:09.215 INFO:teuthology.orchestra.run.smithi149.stdout:(1/23): ceph-common-18.0.0-6170.g277a824d.el8.x 2.8 MB/s | 23 MB 00:07 2023-09-18T20:35:09.292 INFO:teuthology.orchestra.run.smithi149.stdout:(2/23): ceph-selinux-18.0.0-6170.g277a824d.el8. 316 kB/s | 24 kB 00:00 2023-09-18T20:35:09.426 INFO:teuthology.orchestra.run.smithi149.stdout:(3/23): libcephfs2-18.0.0-6170.g277a824d.el8.x8 6.1 MB/s | 827 kB 00:00 2023-09-18T20:35:09.501 INFO:teuthology.orchestra.run.smithi149.stdout:(4/23): libradosstriper1-18.0.0-6170.g277a824d. 6.8 MB/s | 518 kB 00:00 2023-09-18T20:35:09.660 INFO:teuthology.orchestra.run.smithi149.stdout:(5/23): ceph-base-18.0.0-6170.g277a824d.el8.x86 658 kB/s | 5.4 MB 00:08 2023-09-18T20:35:09.827 INFO:teuthology.orchestra.run.smithi149.stdout:(6/23): python3-ceph-argparse-18.0.0-6170.g277a 268 kB/s | 45 kB 00:00 2023-09-18T20:35:09.969 INFO:teuthology.orchestra.run.smithi149.stdout:(7/23): librgw2-18.0.0-6170.g277a824d.el8.x86_6 15 MB/s | 6.9 MB 00:00 2023-09-18T20:35:10.002 INFO:teuthology.orchestra.run.smithi149.stdout:(8/23): python3-ceph-common-18.0.0-6170.g277a82 717 kB/s | 126 kB 00:00 2023-09-18T20:35:10.036 INFO:teuthology.orchestra.run.smithi149.stdout:(9/23): python3-cephfs-18.0.0-6170.g277a824d.el 2.9 MB/s | 196 kB 00:00 2023-09-18T20:35:10.111 INFO:teuthology.orchestra.run.smithi149.stdout:(10/23): python3-rados-18.0.0-6170.g277a824d.el 3.3 MB/s | 371 kB 00:00 2023-09-18T20:35:10.136 INFO:teuthology.orchestra.run.smithi149.stdout:(11/23): python3-rbd-18.0.0-6170.g277a824d.el8. 3.6 MB/s | 364 kB 00:00 2023-09-18T20:35:10.179 INFO:teuthology.orchestra.run.smithi149.stdout:(12/23): python3-rgw-18.0.0-6170.g277a824d.el8. 1.6 MB/s | 107 kB 00:00 2023-09-18T20:35:10.279 INFO:teuthology.orchestra.run.smithi149.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 589 kB/s | 59 kB 00:00 2023-09-18T20:35:10.321 INFO:teuthology.orchestra.run.smithi149.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-09-18T20:35:10.346 INFO:teuthology.orchestra.run.smithi149.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-09-18T20:35:10.405 INFO:teuthology.orchestra.run.smithi149.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 804 kB/s | 47 kB 00:00 2023-09-18T20:35:10.464 INFO:teuthology.orchestra.run.smithi149.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-09-18T20:35:10.505 INFO:teuthology.orchestra.run.smithi149.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-09-18T20:35:10.564 INFO:teuthology.orchestra.run.smithi149.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-09-18T20:35:10.898 INFO:teuthology.orchestra.run.smithi149.stdout:(20/23): ceph-radosgw-18.0.0-6170.g277a824d.el8 1.4 MB/s | 14 MB 00:09 2023-09-18T20:35:10.939 INFO:teuthology.orchestra.run.smithi149.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2023-09-18T20:35:11.190 INFO:teuthology.orchestra.run.smithi149.stdout:(22/23): librbd1-18.0.0-6170.g277a824d.el8.x86_ 13 MB/s | 3.7 MB 00:00 2023-09-18T20:35:11.248 INFO:teuthology.orchestra.run.smithi149.stdout:(23/23): librados2-18.0.0-6170.g277a824d.el8.x8 5.4 MB/s | 3.7 MB 00:00 2023-09-18T20:35:11.251 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:35:11.251 INFO:teuthology.orchestra.run.smithi149.stdout:Total 6.1 MB/s | 62 MB 00:10 2023-09-18T20:35:11.296 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:35:11.341 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:35:11.341 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:35:11.688 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:35:11.692 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:35:12.909 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:35:13.151 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-09-18T20:35:13.368 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-18T20:35:13.781 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-18T20:35:13.951 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librados2-2:18.0.0-6170.g277a824d.el8.x86_64 3/25 2023-09-18T20:35:14.047 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-2:18.0.0-6170.g277a824d.el8.x86_64 3/25 2023-09-18T20:35:14.441 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rados-2:18.0.0-6170.g277a824d.el8.x86_64 4/25 2023-09-18T20:35:14.713 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-09-18T20:35:14.934 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs2-2:18.0.0-6170.g277a824d.el8.x86_64 6/25 2023-09-18T20:35:15.337 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs2-2:18.0.0-6170.g277a824d.el8.x86_64 6/25 2023-09-18T20:35:15.526 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librbd1-2:18.0.0-6170.g277a824d.el8.x86_64 7/25 2023-09-18T20:35:15.632 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-2:18.0.0-6170.g277a824d.el8.x86_64 7/25 2023-09-18T20:35:15.793 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-09-18T20:35:15.847 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-09-18T20:35:16.033 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-09-18T20:35:16.186 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-argparse-2:18.0.0-6170.g277a824d.el8. 10/25 2023-09-18T20:35:16.402 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cephfs-2:18.0.0-6170.g277a824d.el8.x86_64 11/25 2023-09-18T20:35:16.621 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rbd-2:18.0.0-6170.g277a824d.el8.x86_64 12/25 2023-09-18T20:35:16.784 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libradosstriper1-2:18.0.0-6170.g277a824d.el8.x86_6 13/25 2023-09-18T20:35:16.854 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6170.g277a824d.el8.x86_6 13/25 2023-09-18T20:35:17.026 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-09-18T20:35:17.219 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-09-18T20:35:18.025 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-09-18T20:35:18.217 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librgw2-2:18.0.0-6170.g277a824d.el8.x86_64 17/25 2023-09-18T20:35:18.277 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librgw2-2:18.0.0-6170.g277a824d.el8.x86_64 17/25 2023-09-18T20:35:18.561 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rgw-2:18.0.0-6170.g277a824d.el8.x86_64 18/25 2023-09-18T20:35:18.751 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-common-2:18.0.0-6170.g277a824d.el8.x8 19/25 2023-09-18T20:35:22.653 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.0.0-6170.g277a824d.el8.x86_64 20/25 2023-09-18T20:35:22.917 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-common-2:18.0.0-6170.g277a824d.el8.x86_64 20/25 2023-09-18T20:35:23.500 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.0.0-6170.g277a824d.el8.x86_64 20/25 2023-09-18T20:35:23.755 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-base-2:18.0.0-6170.g277a824d.el8.x86_64 21/25 2023-09-18T20:35:23.838 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-base-2:18.0.0-6170.g277a824d.el8.x86_64 21/25 2023-09-18T20:35:23.991 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-selinux-2:18.0.0-6170.g277a824d.el8.x86_64 22/25 2023-09-18T20:35:42.861 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6170.g277a824d.el8.x86_64 22/25 2023-09-18T20:35:42.862 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /sys 2023-09-18T20:35:42.862 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /proc 2023-09-18T20:35:42.862 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2023-09-18T20:35:42.862 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /run 2023-09-18T20:35:42.862 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /mnt 2023-09-18T20:35:42.863 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /var/tmp 2023-09-18T20:35:42.863 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /home 2023-09-18T20:35:42.863 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /tmp 2023-09-18T20:35:42.863 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2023-09-18T20:35:42.863 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:44.277 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-radosgw-2:18.0.0-6170.g277a824d.el8.x86_64 23/25 2023-09-18T20:35:44.298 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6170.g277a824d.el8.x86_64 23/25 2023-09-18T20:35:44.298 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-18T20:35:44.499 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-18T20:35:44.499 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-18T20:35:45.138 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-18T20:35:45.138 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-base-2:18.0.0-6170.g277a824d.el8.x86_64 1/25 2023-09-18T20:35:45.139 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-common-2:18.0.0-6170.g277a824d.el8.x86_64 2/25 2023-09-18T20:35:45.139 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-radosgw-2:18.0.0-6170.g277a824d.el8.x86_64 3/25 2023-09-18T20:35:45.139 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-selinux-2:18.0.0-6170.g277a824d.el8.x86_64 4/25 2023-09-18T20:35:45.139 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs2-2:18.0.0-6170.g277a824d.el8.x86_64 5/25 2023-09-18T20:35:45.140 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libradosstriper1-2:18.0.0-6170.g277a824d.el8.x86_6 6/25 2023-09-18T20:35:45.140 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librgw2-2:18.0.0-6170.g277a824d.el8.x86_64 7/25 2023-09-18T20:35:45.140 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6170.g277a824d.el8. 8/25 2023-09-18T20:35:45.140 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-common-2:18.0.0-6170.g277a824d.el8.x8 9/25 2023-09-18T20:35:45.140 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cephfs-2:18.0.0-6170.g277a824d.el8.x86_64 10/25 2023-09-18T20:35:45.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rados-2:18.0.0-6170.g277a824d.el8.x86_64 11/25 2023-09-18T20:35:45.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rbd-2:18.0.0-6170.g277a824d.el8.x86_64 12/25 2023-09-18T20:35:45.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rgw-2:18.0.0-6170.g277a824d.el8.x86_64 13/25 2023-09-18T20:35:45.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-18T20:35:45.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-18T20:35:45.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-18T20:35:45.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-18T20:35:45.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-09-18T20:35:45.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-18T20:35:45.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-09-18T20:35:45.143 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-09-18T20:35:45.143 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-2:18.0.0-6170.g277a824d.el8.x86_64 22/25 2023-09-18T20:35:45.143 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-18T20:35:45.143 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-2:18.0.0-6170.g277a824d.el8.x86_64 24/25 2023-09-18T20:35:46.570 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-18T20:35:46.570 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:57 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:35:48.976 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:35:49.168 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:49.169 INFO:teuthology.orchestra.run.smithi149.stdout:Upgraded: 2023-09-18T20:35:49.169 INFO:teuthology.orchestra.run.smithi149.stdout: librados2-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.169 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.169 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:35:49.170 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.170 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.170 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.170 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.170 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-18T20:35:49.171 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.171 INFO:teuthology.orchestra.run.smithi149.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-18T20:35:49.171 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-18T20:35:49.171 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.171 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-18T20:35:49.171 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.172 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-18T20:35:49.172 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-18T20:35:49.172 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.172 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.172 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.173 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-18T20:35:49.173 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.173 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.173 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:35:49.173 INFO:teuthology.orchestra.run.smithi149.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-18T20:35:49.173 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:49.174 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:35:49.352 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-test 2023-09-18T20:35:49.702 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:35:50.509 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:01 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:35:51.929 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:35:51.930 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2023-09-18T20:35:51.930 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:35:51.930 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2023-09-18T20:35:51.931 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:35:51.931 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test x86_64 2:18.0.0-6170.g277a824d.el8 ceph 49 M 2023-09-18T20:35:51.931 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:35:51.931 INFO:teuthology.orchestra.run.smithi149.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-09-18T20:35:51.931 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite x86_64 2:18.0.0-6170.g277a824d.el8 ceph 184 k 2023-09-18T20:35:51.932 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-18T20:35:51.932 INFO:teuthology.orchestra.run.smithi149.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-09-18T20:35:51.932 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-18T20:35:51.932 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:51.932 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:35:51.932 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2023-09-18T20:35:51.933 INFO:teuthology.orchestra.run.smithi149.stdout:Install 6 Packages 2023-09-18T20:35:51.933 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:35:51.933 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 50 M 2023-09-18T20:35:51.934 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 223 M 2023-09-18T20:35:51.934 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:35:51.977 INFO:teuthology.orchestra.run.smithi149.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.5 MB/s | 70 kB 00:00 2023-09-18T20:35:52.052 INFO:teuthology.orchestra.run.smithi149.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-09-18T20:35:52.102 INFO:teuthology.orchestra.run.smithi149.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-09-18T20:35:52.153 INFO:teuthology.orchestra.run.smithi149.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 3.9 MB/s | 202 kB 00:00 2023-09-18T20:35:52.245 INFO:teuthology.orchestra.run.smithi149.stdout:(5/6): libcephsqlite-18.0.0-6170.g277a824d.el8. 589 kB/s | 184 kB 00:00 2023-09-18T20:35:54.373 INFO:teuthology.orchestra.run.smithi149.stdout:(6/6): ceph-test-18.0.0-6170.g277a824d.el8.x86_ 20 MB/s | 49 MB 00:02 2023-09-18T20:35:54.373 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:35:54.373 INFO:teuthology.orchestra.run.smithi149.stdout:Total 20 MB/s | 50 MB 00:02 2023-09-18T20:35:54.408 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:35:54.430 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:35:54.430 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:35:54.694 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:35:54.698 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:35:55.141 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:35:55.269 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-09-18T20:35:55.373 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-09-18T20:35:55.579 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-09-18T20:35:55.728 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-09-18T20:35:55.893 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-18T20:35:56.111 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephsqlite-2:18.0.0-6170.g277a824d.el8.x86_64 5/6 2023-09-18T20:36:00.849 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6170.g277a824d.el8.x86_64 5/6 2023-09-18T20:36:01.124 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-test-2:18.0.0-6170.g277a824d.el8.x86_64 6/6 2023-09-18T20:36:01.357 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-test-2:18.0.0-6170.g277a824d.el8.x86_64 6/6 2023-09-18T20:36:01.357 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-test-2:18.0.0-6170.g277a824d.el8.x86_64 1/6 2023-09-18T20:36:01.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephsqlite-2:18.0.0-6170.g277a824d.el8.x86_64 2/6 2023-09-18T20:36:01.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-18T20:36:01.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-09-18T20:36:01.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-18T20:36:02.437 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-09-18T20:36:02.437 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:13 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:04.817 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:36:04.999 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:05.000 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:36:05.000 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:05.000 INFO:teuthology.orchestra.run.smithi149.stdout: jq-1.6-3.el8.x86_64 2023-09-18T20:36:05.000 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:05.001 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-18T20:36:05.001 INFO:teuthology.orchestra.run.smithi149.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-18T20:36:05.001 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-18T20:36:05.001 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:05.001 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:05.144 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph 2023-09-18T20:36:05.482 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:06.309 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:17 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:07.758 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:36:07.762 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================= 2023-09-18T20:36:07.762 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:36:07.762 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================= 2023-09-18T20:36:07.762 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:36:07.762 INFO:teuthology.orchestra.run.smithi149.stdout: ceph x86_64 2:18.0.0-6170.g277a824d.el8 ceph 6.4 k 2023-09-18T20:36:07.763 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:36:07.763 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds x86_64 2:18.0.0-6170.g277a824d.el8 ceph 2.3 M 2023-09-18T20:36:07.763 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr x86_64 2:18.0.0-6170.g277a824d.el8 ceph 1.6 M 2023-09-18T20:36:07.763 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 242 k 2023-09-18T20:36:07.763 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon x86_64 2:18.0.0-6170.g277a824d.el8 ceph 4.6 M 2023-09-18T20:36:07.764 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd x86_64 2:18.0.0-6170.g277a824d.el8 ceph 18 M 2023-09-18T20:36:07.764 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-18T20:36:07.764 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-18T20:36:07.764 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-09-18T20:36:07.765 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-18T20:36:07.765 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-18T20:36:07.765 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-09-18T20:36:07.765 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-18T20:36:07.765 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-18T20:36:07.766 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-18T20:36:07.766 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-18T20:36:07.766 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-18T20:36:07.766 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-18T20:36:07.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-18T20:36:07.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-18T20:36:07.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-09-18T20:36:07.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-09-18T20:36:07.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-18T20:36:07.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-18T20:36:07.768 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-18T20:36:07.768 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-18T20:36:07.768 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-18T20:36:07.768 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-18T20:36:07.768 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-18T20:36:07.769 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-18T20:36:07.769 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-18T20:36:07.769 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-18T20:36:07.769 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:07.769 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:36:07.770 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================= 2023-09-18T20:36:07.770 INFO:teuthology.orchestra.run.smithi149.stdout:Install 32 Packages 2023-09-18T20:36:07.770 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:07.771 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 30 M 2023-09-18T20:36:07.771 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 120 M 2023-09-18T20:36:07.771 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:36:08.080 INFO:teuthology.orchestra.run.smithi149.stdout:(1/32): ceph-18.0.0-6170.g277a824d.el8.x86_64.r 38 kB/s | 6.4 kB 00:00 2023-09-18T20:36:08.463 INFO:teuthology.orchestra.run.smithi149.stdout:(2/32): ceph-mds-18.0.0-6170.g277a824d.el8.x86_ 4.2 MB/s | 2.3 MB 00:00 2023-09-18T20:36:08.505 INFO:teuthology.orchestra.run.smithi149.stdout:(3/32): ceph-mgr-18.0.0-6170.g277a824d.el8.x86_ 2.7 MB/s | 1.6 MB 00:00 2023-09-18T20:36:08.572 INFO:teuthology.orchestra.run.smithi149.stdout:(4/32): ceph-mon-18.0.0-6170.g277a824d.el8.x86_ 9.4 MB/s | 4.6 MB 00:00 2023-09-18T20:36:08.673 INFO:teuthology.orchestra.run.smithi149.stdout:(5/32): ceph-mgr-modules-core-18.0.0-6170.g277a 1.4 MB/s | 242 kB 00:00 2023-09-18T20:36:08.840 INFO:teuthology.orchestra.run.smithi149.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 163 kB/s | 44 kB 00:00 2023-09-18T20:36:09.032 INFO:teuthology.orchestra.run.smithi149.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 901 kB/s | 173 kB 00:00 2023-09-18T20:36:09.048 INFO:teuthology.orchestra.run.smithi149.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 493 kB/s | 185 kB 00:00 2023-09-18T20:36:09.116 INFO:teuthology.orchestra.run.smithi149.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-09-18T20:36:09.182 INFO:teuthology.orchestra.run.smithi149.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2023-09-18T20:36:09.199 INFO:teuthology.orchestra.run.smithi149.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2023-09-18T20:36:09.250 INFO:teuthology.orchestra.run.smithi149.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 729 kB/s | 49 kB 00:00 2023-09-18T20:36:09.283 INFO:teuthology.orchestra.run.smithi149.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 704 kB/s | 59 kB 00:00 2023-09-18T20:36:09.358 INFO:teuthology.orchestra.run.smithi149.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-09-18T20:36:09.375 INFO:teuthology.orchestra.run.smithi149.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-09-18T20:36:09.576 INFO:teuthology.orchestra.run.smithi149.stdout:(16/32): ceph-osd-18.0.0-6170.g277a824d.el8.x86 16 MB/s | 18 MB 00:01 2023-09-18T20:36:09.593 INFO:teuthology.orchestra.run.smithi149.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 80 kB/s | 19 kB 00:00 2023-09-18T20:36:09.610 INFO:teuthology.orchestra.run.smithi149.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 102 kB/s | 24 kB 00:00 2023-09-18T20:36:09.677 INFO:teuthology.orchestra.run.smithi149.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 414 kB/s | 27 kB 00:00 2023-09-18T20:36:09.752 INFO:teuthology.orchestra.run.smithi149.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2023-09-18T20:36:09.777 INFO:teuthology.orchestra.run.smithi149.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2023-09-18T20:36:09.819 INFO:teuthology.orchestra.run.smithi149.stdout:(22/32): python3-tempora-1.14.1-5.el8.noarch.rp 121 kB/s | 29 kB 00:00 2023-09-18T20:36:09.844 INFO:teuthology.orchestra.run.smithi149.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 926 kB/s | 85 kB 00:00 2023-09-18T20:36:09.869 INFO:teuthology.orchestra.run.smithi149.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2023-09-18T20:36:09.928 INFO:teuthology.orchestra.run.smithi149.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2023-09-18T20:36:09.962 INFO:teuthology.orchestra.run.smithi149.stdout:(26/32): python3-cryptography-3.2.1-5.el8.x86_6 4.7 MB/s | 559 kB 00:00 2023-09-18T20:36:09.987 INFO:teuthology.orchestra.run.smithi149.stdout:(27/32): python3-pytz-2017.2-9.el8.noarch.rpm 926 kB/s | 54 kB 00:00 2023-09-18T20:36:10.012 INFO:teuthology.orchestra.run.smithi149.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 774 kB/s | 39 kB 00:00 2023-09-18T20:36:10.038 INFO:teuthology.orchestra.run.smithi149.stdout:(29/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-09-18T20:36:10.063 INFO:teuthology.orchestra.run.smithi149.stdout:(30/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2023-09-18T20:36:10.088 INFO:teuthology.orchestra.run.smithi149.stdout:(31/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.0 MB/s | 103 kB 00:00 2023-09-18T20:36:10.114 INFO:teuthology.orchestra.run.smithi149.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.4 MB/s | 457 kB 00:00 2023-09-18T20:36:10.118 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:36:10.118 INFO:teuthology.orchestra.run.smithi149.stdout:Total 13 MB/s | 30 MB 00:02 2023-09-18T20:36:10.209 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:36:10.247 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:36:10.247 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:36:10.460 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:36:10.464 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:36:11.558 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:36:11.789 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-18T20:36:12.083 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-18T20:36:12.304 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-09-18T20:36:12.527 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-09-18T20:36:12.769 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-09-18T20:36:12.937 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-09-18T20:36:13.168 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-09-18T20:36:13.540 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-09-18T20:36:13.845 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-09-18T20:36:13.999 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-09-18T20:36:14.192 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-09-18T20:36:14.417 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-09-18T20:36:14.630 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-09-18T20:36:14.798 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-09-18T20:36:14.980 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-09-18T20:36:15.170 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-09-18T20:36:15.320 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-09-18T20:36:15.447 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-09-18T20:36:15.687 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-09-18T20:36:15.920 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-09-18T20:36:16.089 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-18T20:36:16.344 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-18T20:36:16.596 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-18T20:36:16.774 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-09-18T20:36:17.083 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-09-18T20:36:17.430 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-18T20:36:17.748 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6170.g277a824d.el8. 27/32 2023-09-18T20:36:18.264 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-2:18.0.0-6170.g277a824d.el8.x86_64 28/32 2023-09-18T20:36:19.858 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6170.g277a824d.el8.x86_64 28/32 2023-09-18T20:36:20.030 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-osd-2:18.0.0-6170.g277a824d.el8.x86_64 29/32 2023-09-18T20:36:20.490 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-osd-2:18.0.0-6170.g277a824d.el8.x86_64 29/32 2023-09-18T20:36:20.616 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mon-2:18.0.0-6170.g277a824d.el8.x86_64 30/32 2023-09-18T20:36:20.835 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mon-2:18.0.0-6170.g277a824d.el8.x86_64 30/32 2023-09-18T20:36:20.991 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mds-2:18.0.0-6170.g277a824d.el8.x86_64 31/32 2023-09-18T20:36:21.010 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mds-2:18.0.0-6170.g277a824d.el8.x86_64 31/32 2023-09-18T20:36:21.105 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-2:18.0.0-6170.g277a824d.el8.x86_64 32/32 2023-09-18T20:36:21.463 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-2:18.0.0-6170.g277a824d.el8.x86_64 32/32 2023-09-18T20:36:21.464 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-2:18.0.0-6170.g277a824d.el8.x86_64 1/32 2023-09-18T20:36:21.464 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mds-2:18.0.0-6170.g277a824d.el8.x86_64 2/32 2023-09-18T20:36:21.464 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-2:18.0.0-6170.g277a824d.el8.x86_64 3/32 2023-09-18T20:36:21.465 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mon-2:18.0.0-6170.g277a824d.el8.x86_64 4/32 2023-09-18T20:36:21.465 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-osd-2:18.0.0-6170.g277a824d.el8.x86_64 5/32 2023-09-18T20:36:21.465 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6170.g277a824d.el8. 6/32 2023-09-18T20:36:21.465 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-09-18T20:36:21.465 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-09-18T20:36:21.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-09-18T20:36:21.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-09-18T20:36:21.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-09-18T20:36:21.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-09-18T20:36:21.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-09-18T20:36:21.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-09-18T20:36:21.467 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-09-18T20:36:21.467 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-09-18T20:36:21.467 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-09-18T20:36:21.467 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-09-18T20:36:21.467 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-09-18T20:36:21.468 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-09-18T20:36:21.468 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-09-18T20:36:21.468 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-09-18T20:36:21.468 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-09-18T20:36:21.468 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-09-18T20:36:21.469 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-09-18T20:36:21.469 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-09-18T20:36:21.469 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-09-18T20:36:21.469 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-09-18T20:36:21.469 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-09-18T20:36:21.470 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-09-18T20:36:21.470 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-09-18T20:36:23.085 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-09-18T20:36:23.086 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:34 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:25.476 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:36:25.662 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:25.663 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:36:25.663 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:25.663 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:25.663 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:25.664 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:36:25.664 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:25.664 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:25.664 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-18T20:36:25.664 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-18T20:36:25.665 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-09-18T20:36:25.665 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-18T20:36:25.665 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-18T20:36:25.665 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-09-18T20:36:25.665 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-18T20:36:25.665 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-18T20:36:25.666 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-18T20:36:25.666 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-18T20:36:25.666 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-18T20:36:25.666 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-18T20:36:25.666 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-18T20:36:25.667 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend-2.6-1.el8.noarch 2023-09-18T20:36:25.667 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-18T20:36:25.667 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-18T20:36:25.667 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-18T20:36:25.667 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-18T20:36:25.667 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-18T20:36:25.668 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-18T20:36:25.668 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-18T20:36:25.668 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-18T20:36:25.668 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-18T20:36:25.668 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-18T20:36:25.668 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-18T20:36:25.669 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-18T20:36:25.669 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:25.669 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:25.860 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-base 2023-09-18T20:36:26.210 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:27.025 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:38 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:28.354 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-base-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:36:28.464 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:36:28.466 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:36:28.466 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:28.808 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install cephadm 2023-09-18T20:36:29.154 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:30.098 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:41 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:31.493 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:36:31.494 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:36:31.495 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:36:31.495 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:36:31.495 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:36:31.495 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 179 k 2023-09-18T20:36:31.495 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:31.495 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:36:31.496 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:36:31.496 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:36:31.496 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:31.496 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 179 k 2023-09-18T20:36:31.496 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 722 k 2023-09-18T20:36:31.497 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:36:31.803 INFO:teuthology.orchestra.run.smithi149.stdout:cephadm-18.0.0-6170.g277a824d.el8.noarch.rpm 581 kB/s | 179 kB 00:00 2023-09-18T20:36:31.804 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:36:31.804 INFO:teuthology.orchestra.run.smithi149.stdout:Total 578 kB/s | 179 kB 00:00 2023-09-18T20:36:31.805 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:36:31.813 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:36:31.813 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:36:31.815 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:36:31.819 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:36:31.860 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:36:33.595 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:36:33.731 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephadm-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:36:33.907 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:36:34.967 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephadm-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:36:34.967 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:45 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:37.284 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:36:37.464 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:37.464 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:36:37.465 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:36:37.465 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:37.465 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:37.643 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-immutable-object-cache 2023-09-18T20:36:38.001 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:38.804 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:49 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:40.206 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:36:40.207 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:36:40.207 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2023-09-18T20:36:40.207 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:36:40.208 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:36:40.208 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6170.g277a824d.el8 ceph 167 k 2023-09-18T20:36:40.208 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:40.208 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:36:40.208 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:36:40.209 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:36:40.209 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:40.209 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 167 k 2023-09-18T20:36:40.209 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 512 k 2023-09-18T20:36:40.209 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:36:40.584 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-immutable-object-cache-18.0.0-6170.g277a82 444 kB/s | 167 kB 00:00 2023-09-18T20:36:40.585 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:36:40.585 INFO:teuthology.orchestra.run.smithi149.stdout:Total 443 kB/s | 167 kB 00:00 2023-09-18T20:36:40.585 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:36:40.596 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:36:40.596 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:36:40.648 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:36:40.651 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:36:40.768 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:36:40.913 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6170.g277a824d. 1/1 2023-09-18T20:36:41.283 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6170.g277a824d. 1/1 2023-09-18T20:36:42.353 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6170.g277a824d. 1/1 2023-09-18T20:36:42.353 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:53 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:44.682 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:36:44.866 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:44.866 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:36:44.866 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:36:44.867 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:44.867 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:45.017 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr 2023-09-18T20:36:45.363 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:46.172 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:57 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:47.450 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:36:47.558 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:36:47.559 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:36:47.559 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:47.692 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-dashboard 2023-09-18T20:36:48.035 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:48.883 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:59 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:50.350 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:36:50.351 INFO:teuthology.orchestra.run.smithi149.stdout:=================================================================================================== 2023-09-18T20:36:50.352 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:36:50.352 INFO:teuthology.orchestra.run.smithi149.stdout:=================================================================================================== 2023-09-18T20:36:50.352 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:36:50.352 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 1.7 M 2023-09-18T20:36:50.352 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:36:50.353 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 23 k 2023-09-18T20:36:50.353 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 15 k 2023-09-18T20:36:50.353 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-18T20:36:50.353 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-18T20:36:50.353 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-18T20:36:50.354 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:50.354 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:36:50.354 INFO:teuthology.orchestra.run.smithi149.stdout:=================================================================================================== 2023-09-18T20:36:50.354 INFO:teuthology.orchestra.run.smithi149.stdout:Install 6 Packages 2023-09-18T20:36:50.354 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:50.355 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 2.1 M 2023-09-18T20:36:50.355 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 8.5 M 2023-09-18T20:36:50.356 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:36:51.969 INFO:teuthology.orchestra.run.smithi149.stdout:(1/6): ceph-mgr-dashboard-18.0.0-6170.g277a824d 1.2 MB/s | 1.7 MB 00:01 2023-09-18T20:36:51.997 INFO:teuthology.orchestra.run.smithi149.stdout:(2/6): ceph-prometheus-alerts-18.0.0-6170.g277a 9.9 kB/s | 15 kB 00:01 2023-09-18T20:36:52.023 INFO:teuthology.orchestra.run.smithi149.stdout:(3/6): ceph-grafana-dashboards-18.0.0-6170.g277 15 kB/s | 23 kB 00:01 2023-09-18T20:36:52.090 INFO:teuthology.orchestra.run.smithi149.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 639 kB/s | 43 kB 00:00 2023-09-18T20:36:52.373 INFO:teuthology.orchestra.run.smithi149.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 83 kB/s | 34 kB 00:00 2023-09-18T20:36:52.532 INFO:teuthology.orchestra.run.smithi149.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 368 kB/s | 196 kB 00:00 2023-09-18T20:36:52.534 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:36:52.534 INFO:teuthology.orchestra.run.smithi149.stdout:Total 963 kB/s | 2.1 MB 00:02 2023-09-18T20:36:52.548 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:36:52.558 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:36:52.559 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:36:52.620 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:36:52.624 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:36:52.900 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:36:53.070 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-18T20:36:53.258 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-18T20:36:53.446 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-18T20:36:53.617 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6170.g277a824d.el8.n 4/6 2023-09-18T20:36:53.953 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6170.g277a824d.el8. 5/6 2023-09-18T20:36:54.202 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6170.g277a824d.el8.noarc 6/6 2023-09-18T20:36:54.443 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6170.g277a824d.el8.noarc 6/6 2023-09-18T20:36:54.443 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6170.g277a824d.el8. 1/6 2023-09-18T20:36:54.444 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6170.g277a824d.el8.noarc 2/6 2023-09-18T20:36:54.444 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6170.g277a824d.el8.n 3/6 2023-09-18T20:36:54.444 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-18T20:36:54.444 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-18T20:36:55.487 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-18T20:36:55.487 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:06 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:36:57.903 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:36:58.085 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:58.085 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:36:58.086 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:36:58.086 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:36:58.086 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:36:58.086 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-18T20:36:58.086 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-18T20:36:58.087 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-18T20:36:58.087 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:36:58.087 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:36:58.237 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-18T20:36:58.585 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:36:59.419 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:10 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:00.801 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:37:00.803 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================================= 2023-09-18T20:37:00.803 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:37:00.803 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================================= 2023-09-18T20:37:00.803 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:37:00.803 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 7.4 M 2023-09-18T20:37:00.804 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:37:00.804 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-09-18T20:37:00.804 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-09-18T20:37:00.804 INFO:teuthology.orchestra.run.smithi149.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-18T20:37:00.804 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-09-18T20:37:00.805 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-18T20:37:00.805 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-18T20:37:00.805 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-09-18T20:37:00.805 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:00.806 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:37:00.806 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================================= 2023-09-18T20:37:00.806 INFO:teuthology.orchestra.run.smithi149.stdout:Install 8 Packages 2023-09-18T20:37:00.806 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:00.807 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 35 M 2023-09-18T20:37:00.807 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 210 M 2023-09-18T20:37:00.807 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:37:00.921 INFO:teuthology.orchestra.run.smithi149.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 1.4 MB/s | 171 kB 00:00 2023-09-18T20:37:00.954 INFO:teuthology.orchestra.run.smithi149.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 4.2 MB/s | 644 kB 00:00 2023-09-18T20:37:01.097 INFO:teuthology.orchestra.run.smithi149.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2023-09-18T20:37:01.177 INFO:teuthology.orchestra.run.smithi149.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.8 MB/s | 225 kB 00:00 2023-09-18T20:37:01.347 INFO:teuthology.orchestra.run.smithi149.stdout:(5/8): python3-scipy-1.0.0-21.module+el8.5.0+10 32 MB/s | 14 MB 00:00 2023-09-18T20:37:01.439 INFO:teuthology.orchestra.run.smithi149.stdout:(6/8): ceph-mgr-diskprediction-local-18.0.0-617 12 MB/s | 7.4 MB 00:00 2023-09-18T20:37:01.548 INFO:teuthology.orchestra.run.smithi149.stdout:(7/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2023-09-18T20:37:01.614 INFO:teuthology.orchestra.run.smithi149.stdout:(8/8): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-09-18T20:37:01.615 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:37:01.616 INFO:teuthology.orchestra.run.smithi149.stdout:Total 44 MB/s | 35 MB 00:00 2023-09-18T20:37:01.855 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:37:01.873 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:37:01.873 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:37:02.167 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:37:02.171 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:37:02.922 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:37:03.075 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-09-18T20:37:03.217 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-09-18T20:37:03.400 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-09-18T20:37:04.014 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-09-18T20:37:04.175 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-09-18T20:37:04.772 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-09-18T20:37:04.900 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-09-18T20:37:05.598 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-09-18T20:37:05.887 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-18T20:37:06.075 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-18T20:37:07.837 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-18T20:37:09.335 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-09-18T20:37:09.491 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6170.g277a824 8/8 2023-09-18T20:37:09.759 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6170.g277a824 8/8 2023-09-18T20:37:09.760 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6170.g277a824 1/8 2023-09-18T20:37:09.760 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-09-18T20:37:09.760 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-09-18T20:37:09.760 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-09-18T20:37:09.761 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-18T20:37:09.761 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-18T20:37:09.761 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-09-18T20:37:10.858 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-09-18T20:37:10.859 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:21 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:13.168 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:37:13.352 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:13.352 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:37:13.353 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:37:13.353 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-09-18T20:37:13.353 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-09-18T20:37:13.353 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-0.3.15-3.el8.x86_64 2023-09-18T20:37:13.353 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-09-18T20:37:13.354 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-18T20:37:13.354 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-18T20:37:13.354 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-09-18T20:37:13.354 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:13.355 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:37:13.521 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-rook 2023-09-18T20:37:13.869 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:37:14.687 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:25 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:16.096 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:37:16.097 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2023-09-18T20:37:16.098 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:37:16.098 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2023-09-18T20:37:16.098 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:37:16.099 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 47 k 2023-09-18T20:37:16.099 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:37:16.099 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-18T20:37:16.099 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-18T20:37:16.100 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-18T20:37:16.100 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-18T20:37:16.100 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-18T20:37:16.100 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-18T20:37:16.100 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-18T20:37:16.101 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-18T20:37:16.101 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-18T20:37:16.101 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-18T20:37:16.101 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-18T20:37:16.101 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-18T20:37:16.102 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:16.102 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:37:16.102 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2023-09-18T20:37:16.102 INFO:teuthology.orchestra.run.smithi149.stdout:Install 13 Packages 2023-09-18T20:37:16.103 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:16.103 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 1.9 M 2023-09-18T20:37:16.103 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 25 M 2023-09-18T20:37:16.104 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:37:16.311 INFO:teuthology.orchestra.run.smithi149.stdout:(1/13): ceph-mgr-rook-18.0.0-6170.g277a824d.el8 266 kB/s | 47 kB 00:00 2023-09-18T20:37:16.570 INFO:teuthology.orchestra.run.smithi149.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 37 kB/s | 16 kB 00:00 2023-09-18T20:37:16.603 INFO:teuthology.orchestra.run.smithi149.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 70 kB/s | 33 kB 00:00 2023-09-18T20:37:16.829 INFO:teuthology.orchestra.run.smithi149.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 285 kB/s | 64 kB 00:00 2023-09-18T20:37:16.854 INFO:teuthology.orchestra.run.smithi149.stdout:(5/13): python3-google-auth-1.1.1-10.el8.noarch 150 kB/s | 81 kB 00:00 2023-09-18T20:37:16.921 INFO:teuthology.orchestra.run.smithi149.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 640 kB/s | 43 kB 00:00 2023-09-18T20:37:16.971 INFO:teuthology.orchestra.run.smithi149.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2023-09-18T20:37:16.996 INFO:teuthology.orchestra.run.smithi149.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 365 kB/s | 61 kB 00:00 2023-09-18T20:37:17.022 INFO:teuthology.orchestra.run.smithi149.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-09-18T20:37:17.080 INFO:teuthology.orchestra.run.smithi149.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.4 MB/s | 126 kB 00:00 2023-09-18T20:37:17.097 INFO:teuthology.orchestra.run.smithi149.stdout:(11/13): python3-jsonpatch-1.21-2.el8.noarch.rp 264 kB/s | 27 kB 00:00 2023-09-18T20:37:17.122 INFO:teuthology.orchestra.run.smithi149.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 478 kB/s | 20 kB 00:00 2023-09-18T20:37:17.273 INFO:teuthology.orchestra.run.smithi149.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2023-09-18T20:37:17.276 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:37:17.276 INFO:teuthology.orchestra.run.smithi149.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2023-09-18T20:37:17.342 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:37:17.359 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:37:17.359 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:37:17.432 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:37:17.436 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:37:17.907 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:37:18.125 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-18T20:37:18.334 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-18T20:37:18.504 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-18T20:37:18.647 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-09-18T20:37:18.876 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-09-18T20:37:19.077 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-18T20:37:19.267 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-18T20:37:19.455 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-18T20:37:19.637 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-18T20:37:19.826 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-18T20:37:20.905 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-18T20:37:21.123 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-18T20:37:21.273 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-rook-2:18.0.0-6170.g277a824d.el8.noarch 13/13 2023-09-18T20:37:21.518 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6170.g277a824d.el8.noarch 13/13 2023-09-18T20:37:21.519 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6170.g277a824d.el8.noarch 1/13 2023-09-18T20:37:21.519 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-18T20:37:21.519 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-18T20:37:21.519 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-18T20:37:21.519 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-18T20:37:21.520 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-18T20:37:21.520 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-18T20:37:21.520 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-09-18T20:37:21.520 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-09-18T20:37:21.520 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-18T20:37:21.521 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-09-18T20:37:21.521 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-09-18T20:37:22.689 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-09-18T20:37:22.689 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:33 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:25.061 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:37:25.246 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:25.246 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:37:25.247 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:37:25.247 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-18T20:37:25.247 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-18T20:37:25.247 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-18T20:37:25.247 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-18T20:37:25.248 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-18T20:37:25.248 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-18T20:37:25.248 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-18T20:37:25.248 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-18T20:37:25.248 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-18T20:37:25.249 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-18T20:37:25.249 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-18T20:37:25.249 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-18T20:37:25.249 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:25.249 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:37:25.405 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-cephadm 2023-09-18T20:37:25.759 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:37:26.593 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:37 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:28.058 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:37:28.061 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================================== 2023-09-18T20:37:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:37:28.061 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================================== 2023-09-18T20:37:28.062 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:37:28.062 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 132 k 2023-09-18T20:37:28.062 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:37:28.062 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-18T20:37:28.062 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-18T20:37:28.063 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-18T20:37:28.063 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-18T20:37:28.063 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:28.063 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:37:28.063 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================================== 2023-09-18T20:37:28.063 INFO:teuthology.orchestra.run.smithi149.stdout:Install 5 Packages 2023-09-18T20:37:28.064 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:28.064 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 6.0 M 2023-09-18T20:37:28.064 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 26 M 2023-09-18T20:37:28.064 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:37:28.579 INFO:teuthology.orchestra.run.smithi149.stdout:(1/5): ceph-mgr-cephadm-18.0.0-6170.g277a824d.e 627 kB/s | 132 kB 00:00 2023-09-18T20:37:28.605 INFO:teuthology.orchestra.run.smithi149.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 255 kB/s | 60 kB 00:00 2023-09-18T20:37:28.680 INFO:teuthology.orchestra.run.smithi149.stdout:(3/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.2 MB/s | 538 kB 00:00 2023-09-18T20:37:28.780 INFO:teuthology.orchestra.run.smithi149.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2023-09-18T20:37:28.805 INFO:teuthology.orchestra.run.smithi149.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.1 MB/s | 476 kB 00:00 2023-09-18T20:37:28.807 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:37:28.807 INFO:teuthology.orchestra.run.smithi149.stdout:Total 8.0 MB/s | 6.0 MB 00:00 2023-09-18T20:37:28.862 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:37:28.873 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:37:28.873 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:37:28.953 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:37:28.957 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:37:29.956 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:37:30.316 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-09-18T20:37:30.514 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-09-18T20:37:30.857 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-18T20:37:31.119 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-18T20:37:31.313 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6170.g277a824d.el8.noarch 5/5 2023-09-18T20:37:31.557 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6170.g277a824d.el8.noarch 5/5 2023-09-18T20:37:31.558 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6170.g277a824d.el8.noarch 1/5 2023-09-18T20:37:31.558 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-18T20:37:31.558 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-18T20:37:31.558 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-18T20:37:32.621 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-09-18T20:37:32.622 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:43 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:35.039 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:37:35.222 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:35.222 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:37:35.222 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:37:35.223 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-18T20:37:35.223 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-18T20:37:35.223 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-18T20:37:35.223 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-18T20:37:35.223 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:35.224 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:37:35.339 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-fuse 2023-09-18T20:37:35.699 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:37:36.504 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:47 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:37.891 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:37:37.892 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:37.892 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:37:37.892 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:37.892 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:37:37.892 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse 2023-09-18T20:37:37.893 INFO:teuthology.orchestra.run.smithi149.stdout: x86_64 2:18.0.0-6170.g277a824d.el8 ceph 907 k 2023-09-18T20:37:37.893 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:37:37.893 INFO:teuthology.orchestra.run.smithi149.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-18T20:37:37.893 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:37.893 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:37:37.894 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:37.894 INFO:teuthology.orchestra.run.smithi149.stdout:Install 2 Packages 2023-09-18T20:37:37.894 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:37.894 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 990 k 2023-09-18T20:37:37.894 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 3.1 M 2023-09-18T20:37:37.894 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:37:37.964 INFO:teuthology.orchestra.run.smithi149.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-09-18T20:37:38.172 INFO:teuthology.orchestra.run.smithi149.stdout:(2/2): ceph-fuse-18.0.0-6170.g277a824d.el8.x86_ 3.2 MB/s | 907 kB 00:00 2023-09-18T20:37:38.173 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:37:38.173 INFO:teuthology.orchestra.run.smithi149.stdout:Total 3.4 MB/s | 990 kB 00:00 2023-09-18T20:37:38.184 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:37:38.206 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:37:38.206 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:37:38.280 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:37:38.284 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:37:38.437 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:37:38.707 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-09-18T20:37:38.851 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 2/2 2023-09-18T20:37:39.193 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 2/2 2023-09-18T20:37:39.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 1/2 2023-09-18T20:37:40.310 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-09-18T20:37:40.311 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:51 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:42.740 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:37:42.921 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:42.921 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:37:42.922 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-09-18T20:37:42.922 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:42.922 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:37:43.085 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-volume 2023-09-18T20:37:43.443 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:37:44.271 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:55 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:45.666 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:37:45.667 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:45.668 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:37:45.668 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:45.668 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:37:45.668 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume noarch 2:18.0.0-6170.g277a824d.el8 ceph-noarch 261 k 2023-09-18T20:37:45.668 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:45.668 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:37:45.669 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:45.669 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:37:45.669 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:45.669 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 261 k 2023-09-18T20:37:45.669 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 1.3 M 2023-09-18T20:37:45.669 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:37:46.004 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-volume-18.0.0-6170.g277a824d.el8.noarch.rp 777 kB/s | 261 kB 00:00 2023-09-18T20:37:46.005 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:37:46.005 INFO:teuthology.orchestra.run.smithi149.stdout:Total 775 kB/s | 261 kB 00:00 2023-09-18T20:37:46.005 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:37:46.015 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:37:46.016 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:37:46.033 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:37:46.037 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:37:46.202 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:37:46.736 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-volume-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:37:47.098 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-volume-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:37:48.188 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-volume-2:18.0.0-6170.g277a824d.el8.noarch 1/1 2023-09-18T20:37:48.188 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:59 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:50.502 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:37:50.683 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:50.684 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:37:50.684 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume-2:18.0.0-6170.g277a824d.el8.noarch 2023-09-18T20:37:50.684 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:50.684 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:37:50.837 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librados-devel 2023-09-18T20:37:51.196 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:37:52.009 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:03 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:53.398 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:37:53.399 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:53.399 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2023-09-18T20:37:53.399 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:53.399 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:37:53.400 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel x86_64 2:18.0.0-6170.g277a824d.el8 ceph 139 k 2023-09-18T20:37:53.400 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:53.400 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:37:53.400 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:37:53.401 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:37:53.401 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:53.401 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 139 k 2023-09-18T20:37:53.401 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 513 k 2023-09-18T20:37:53.401 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:37:53.711 INFO:teuthology.orchestra.run.smithi149.stdout:librados-devel-18.0.0-6170.g277a824d.el8.x86_64 447 kB/s | 139 kB 00:00 2023-09-18T20:37:53.711 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:37:53.712 INFO:teuthology.orchestra.run.smithi149.stdout:Total 445 kB/s | 139 kB 00:00 2023-09-18T20:37:53.712 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:37:53.717 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:37:53.717 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:37:53.772 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:37:53.776 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:37:53.888 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:37:54.032 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librados-devel-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:37:54.274 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados-devel-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:37:55.365 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados-devel-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:37:55.365 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:06 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:37:57.766 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:37:57.946 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:57.947 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:37:57.947 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:37:57.947 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:37:57.947 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:37:58.113 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install libcephfs2 2023-09-18T20:37:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:37:59.282 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:10 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:00.629 INFO:teuthology.orchestra.run.smithi149.stdout:Package libcephfs2-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:00.739 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:00.740 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:00.740 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:00.874 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install libcephfs-devel 2023-09-18T20:38:01.230 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:02.058 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:13 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:03.467 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:03.468 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:03.468 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2023-09-18T20:38:03.468 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:03.469 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:38:03.469 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel x86_64 2:18.0.0-6170.g277a824d.el8 ceph 31 k 2023-09-18T20:38:03.469 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:03.469 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:38:03.469 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:03.470 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:38:03.470 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:03.470 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 31 k 2023-09-18T20:38:03.470 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 139 k 2023-09-18T20:38:03.470 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:38:03.765 INFO:teuthology.orchestra.run.smithi149.stdout:libcephfs-devel-18.0.0-6170.g277a824d.el8.x86_6 105 kB/s | 31 kB 00:00 2023-09-18T20:38:03.766 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:38:03.766 INFO:teuthology.orchestra.run.smithi149.stdout:Total 104 kB/s | 31 kB 00:00 2023-09-18T20:38:03.766 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:38:03.770 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:38:03.770 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:38:03.781 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:38:03.785 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:38:03.839 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:38:04.036 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs-devel-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:04.243 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:05.263 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs-devel-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:05.263 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:16 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:07.554 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:38:07.737 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:07.737 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:38:07.737 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:38:07.738 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:07.738 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:07.902 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librados2 2023-09-18T20:38:08.256 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:09.098 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:20 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:10.381 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados2-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:10.488 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:10.489 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:10.490 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:10.635 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librbd1 2023-09-18T20:38:10.991 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:11.835 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:22 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:13.122 INFO:teuthology.orchestra.run.smithi149.stdout:Package librbd1-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:13.230 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:13.232 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:13.232 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:13.368 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rados 2023-09-18T20:38:13.722 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:14.532 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:25 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:15.887 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rados-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:15.997 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:15.998 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:15.998 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:16.130 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rgw 2023-09-18T20:38:16.485 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:17.299 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:28 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:18.581 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rgw-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:18.691 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:18.692 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:18.693 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:18.829 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-cephfs 2023-09-18T20:38:19.188 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:20.025 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:31 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:21.310 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-cephfs-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:21.418 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:21.419 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:21.420 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:21.543 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rbd 2023-09-18T20:38:21.899 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:22.803 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:33 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:24.094 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rbd-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:38:24.201 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:24.202 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-09-18T20:38:24.202 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:24.334 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-fuse 2023-09-18T20:38:24.690 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:25.498 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:36 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:26.956 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:26.957 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:26.957 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2023-09-18T20:38:26.957 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:26.958 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:38:26.958 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse x86_64 2:18.0.0-6170.g277a824d.el8 ceph 94 k 2023-09-18T20:38:26.958 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:26.958 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:38:26.958 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:26.959 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:38:26.959 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:26.959 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 94 k 2023-09-18T20:38:26.959 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 276 k 2023-09-18T20:38:26.959 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:38:27.159 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-fuse-18.0.0-6170.g277a824d.el8.x86_64.rpm 467 kB/s | 94 kB 00:00 2023-09-18T20:38:27.160 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:38:27.160 INFO:teuthology.orchestra.run.smithi149.stdout:Total 464 kB/s | 94 kB 00:00 2023-09-18T20:38:27.160 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:38:27.171 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:38:27.171 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:38:27.224 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:38:27.228 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:38:27.345 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:38:27.510 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:27.757 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:28.843 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:28.843 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:39 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:31.245 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:38:31.428 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:31.428 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:38:31.428 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:38:31.429 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:31.429 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:31.603 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-mirror 2023-09-18T20:38:31.961 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:32.844 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:43 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:34.230 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:34.231 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:34.231 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2023-09-18T20:38:34.231 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:34.231 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:38:34.232 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror x86_64 2:18.0.0-6170.g277a824d.el8 ceph 3.6 M 2023-09-18T20:38:34.232 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:34.232 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:38:34.232 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:34.232 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:38:34.233 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:34.233 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 3.6 M 2023-09-18T20:38:34.233 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 17 M 2023-09-18T20:38:34.233 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:38:34.612 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-mirror-18.0.0-6170.g277a824d.el8.x86_64.rpm 9.4 MB/s | 3.6 MB 00:00 2023-09-18T20:38:34.612 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:38:34.613 INFO:teuthology.orchestra.run.smithi149.stdout:Total 9.4 MB/s | 3.6 MB 00:00 2023-09-18T20:38:34.613 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:38:34.625 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:38:34.625 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:38:34.692 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:38:34.695 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:38:35.146 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:38:35.311 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-mirror-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:35.676 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:36.779 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-mirror-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:36.779 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:47 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:39.130 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:38:39.315 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:39.316 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:38:39.316 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:38:39.316 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:39.316 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:39.463 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-nbd 2023-09-18T20:38:39.823 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:38:40.668 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:51 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:42.048 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:38:42.049 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:42.049 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2023-09-18T20:38:42.049 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:42.049 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:38:42.049 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd x86_64 2:18.0.0-6170.g277a824d.el8 ceph 179 k 2023-09-18T20:38:42.050 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:42.050 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:38:42.050 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:38:42.050 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2023-09-18T20:38:42.050 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:42.051 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 179 k 2023-09-18T20:38:42.051 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 564 k 2023-09-18T20:38:42.051 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:38:42.265 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-nbd-18.0.0-6170.g277a824d.el8.x86_64.rpm 832 kB/s | 179 kB 00:00 2023-09-18T20:38:42.266 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:38:42.266 INFO:teuthology.orchestra.run.smithi149.stdout:Total 827 kB/s | 179 kB 00:00 2023-09-18T20:38:42.266 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:38:42.277 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:38:42.277 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:38:42.331 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:38:42.335 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:38:42.450 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:38:42.599 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-nbd-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:42.843 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:43.930 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-nbd-2:18.0.0-6170.g277a824d.el8.x86_64 1/1 2023-09-18T20:38:43.930 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:03:54 ago on Mon 18 Sep 2023 08:34:49 PM UTC. 2023-09-18T20:38:46.269 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:38:46.449 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:46.449 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:38:46.449 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd-2:18.0.0-6170.g277a824d.el8.x86_64 2023-09-18T20:38:46.450 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:46.450 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:38:46.601 DEBUG:teuthology.parallel:result is None 2023-09-18T20:38:46.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=277a824d757a39499f35213b338ab8a8372df0f6 2023-09-18T20:38:46.820 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-18T20:38:46.843 INFO:teuthology.orchestra.run.smithi149.stdout:18.0.0-6170.g277a824d.el8 2023-09-18T20:38:46.844 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6170.g277a824d.el8 2023-09-18T20:38:46.844 INFO:teuthology.task.install:The correct ceph version 18.0.0-6170.g277a824d is installed. 2023-09-18T20:38:46.845 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-18T20:38:46.846 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:46.846 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-18T20:38:46.911 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-18T20:38:46.911 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:46.912 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/daemon-helper 2023-09-18T20:38:46.977 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-18T20:38:47.043 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-18T20:38:47.044 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:47.044 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-18T20:38:47.109 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-18T20:38:47.176 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-18T20:38:47.177 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:47.177 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/stdin-killer 2023-09-18T20:38:47.243 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-18T20:38:47.309 INFO:teuthology.run_tasks:Running task ceph... 2023-09-18T20:38:47.376 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-18T20:38:47.377 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /var/log/ceph 2023-09-18T20:38:47.403 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-18T20:38:47.403 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-18T20:38:47.469 INFO:tasks.ceph:Creating extra log directories... 2023-09-18T20:38:47.470 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-18T20:38:47.544 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-18T20:38:47.544 INFO:tasks.ceph:config {'conf': {'client': {'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug_rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-18T20:38:47.544 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734', 'branch': 'wip-ajarr-repeated-rbd-client-blocklisting', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7397734', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-ajarr-repeated-rbd-client-blocklisting'}, 'ceph': {'conf': {'client': {'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug_rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ajarr-repeated-rbd-client-blocklisting-2', 'sha1': 'daad63c9e55979797c86b6218776b5cf580dafc3'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1851, 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-ajarr-repeated-rbd-client-blocklisting-2', 'suite_path': '/home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ajarr/ceph', 'suite_sha1': 'daad63c9e55979797c86b6218776b5cf580dafc3', 'targets': {'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-18_20:19:35', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1394'} 2023-09-18T20:38:47.545 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-18T20:38:47.593 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/run/ceph 2023-09-18T20:38:47.659 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:47.660 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2023-09-18T20:38:47.716 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-18T20:38:47.717 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2023-09-18T20:38:47.773 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-18T20:38:47.774 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-18T20:38:47.774 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 383433 Links: 1 2023-09-18T20:38:47.774 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-18T20:38:47.774 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-09-18T20:38:47.774 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-09-18 20:38:35.470641094 +0000 2023-09-18T20:38:47.775 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-09-18 20:34:04.055832675 +0000 2023-09-18T20:38:47.775 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-09-18 20:34:04.055832675 +0000 2023-09-18T20:38:47.775 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-09-18T20:38:47.775 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-18T20:38:47.839 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-09-18T20:38:47.839 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-09-18T20:38:47.839 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000262431 s, 2.0 MB/s 2023-09-18T20:38:47.840 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-18T20:38:47.898 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2023-09-18T20:38:47.955 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-18T20:38:47.956 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-18T20:38:47.956 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 381730 Links: 1 2023-09-18T20:38:47.956 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-18T20:38:47.956 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-09-18T20:38:47.956 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-09-18 20:38:35.470641094 +0000 2023-09-18T20:38:47.957 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-09-18 20:34:04.690819949 +0000 2023-09-18T20:38:47.957 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-09-18 20:34:04.690819949 +0000 2023-09-18T20:38:47.957 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-09-18T20:38:47.957 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-18T20:38:48.020 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-09-18T20:38:48.021 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-09-18T20:38:48.021 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000203118 s, 2.5 MB/s 2023-09-18T20:38:48.022 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-18T20:38:48.079 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2023-09-18T20:38:48.137 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-18T20:38:48.137 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-18T20:38:48.137 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 390471 Links: 1 2023-09-18T20:38:48.137 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-18T20:38:48.138 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-09-18T20:38:48.138 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-09-18 20:38:35.470641094 +0000 2023-09-18T20:38:48.138 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-09-18 20:34:05.202809688 +0000 2023-09-18T20:38:48.138 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-09-18 20:34:05.202809688 +0000 2023-09-18T20:38:48.138 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-09-18T20:38:48.139 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-18T20:38:48.201 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-09-18T20:38:48.201 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-09-18T20:38:48.201 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000223749 s, 2.3 MB/s 2023-09-18T20:38:48.202 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-18T20:38:48.260 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2023-09-18T20:38:48.317 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-18T20:38:48.317 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-18T20:38:48.318 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 392652 Links: 1 2023-09-18T20:38:48.318 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-18T20:38:48.318 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-09-18T20:38:48.318 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-09-18 20:38:35.471641074 +0000 2023-09-18T20:38:48.319 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-09-18 20:34:05.708799548 +0000 2023-09-18T20:38:48.319 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-09-18 20:34:05.708799548 +0000 2023-09-18T20:38:48.319 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-09-18T20:38:48.319 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-18T20:38:48.382 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-09-18T20:38:48.382 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-09-18T20:38:48.382 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000223652 s, 2.3 MB/s 2023-09-18T20:38:48.383 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-18T20:38:48.441 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-18T20:38:48.441 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-18T20:38:48.442 INFO:tasks.ceph:Generating config... 2023-09-18T20:38:48.443 INFO:tasks.ceph:[client] rbd default features = 1 2023-09-18T20:38:48.443 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-18T20:38:48.443 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-18T20:38:48.444 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-18T20:38:48.444 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-18T20:38:48.444 INFO:tasks.ceph:[mgr] debug_rbd = 20 2023-09-18T20:38:48.444 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-18T20:38:48.445 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-18T20:38:48.445 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-18T20:38:48.445 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-18T20:38:48.445 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-09-18T20:38:48.446 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-18T20:38:48.446 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-18T20:38:48.446 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-09-18T20:38:48.447 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-09-18T20:38:48.447 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-18T20:38:48.447 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-18T20:38:48.447 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-09-18T20:38:48.448 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-18T20:38:48.448 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-18T20:38:48.448 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-18T20:38:48.449 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-18T20:38:48.449 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-18T20:38:48.449 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-18T20:38:48.450 INFO:tasks.ceph:Setting up mon.a... 2023-09-18T20:38:48.450 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-18T20:38:48.548 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/ceph.keyring 2023-09-18T20:38:48.551 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-18T20:38:48.596 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-18T20:38:48.622 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.149')] 2023-09-18T20:38:48.622 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.149', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug_rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default features': 1}, 'mon.a': {}} 2023-09-18T20:38:48.623 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:48.624 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-18T20:38:48.678 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.149 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:38:48.759 INFO:teuthology.orchestra.run.smithi149.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:38:48.760 INFO:teuthology.orchestra.run.smithi149.stdout:monmaptool: generated fsid 6b7573aa-3580-401f-a189-159cf58e6075 2023-09-18T20:38:48.760 INFO:teuthology.orchestra.run.smithi149.stdout:setting min_mon_release = pacific 2023-09-18T20:38:48.760 INFO:teuthology.orchestra.run.smithi149.stdout:epoch 0 2023-09-18T20:38:48.760 INFO:teuthology.orchestra.run.smithi149.stdout:fsid 6b7573aa-3580-401f-a189-159cf58e6075 2023-09-18T20:38:48.761 INFO:teuthology.orchestra.run.smithi149.stdout:last_changed 2023-09-18T20:38:48.760288+0000 2023-09-18T20:38:48.761 INFO:teuthology.orchestra.run.smithi149.stdout:created 2023-09-18T20:38:48.760288+0000 2023-09-18T20:38:48.761 INFO:teuthology.orchestra.run.smithi149.stdout:min_mon_release 16 (pacific) 2023-09-18T20:38:48.761 INFO:teuthology.orchestra.run.smithi149.stdout:election_strategy: 1 2023-09-18T20:38:48.761 INFO:teuthology.orchestra.run.smithi149.stdout:0: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.a 2023-09-18T20:38:48.761 INFO:teuthology.orchestra.run.smithi149.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-09-18T20:38:48.763 DEBUG:teuthology.orchestra.run.smithi149:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-18T20:38:48.819 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6b7573aa-3580-401f-a189-159cf58e6075... 2023-09-18T20:38:48.820 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-18T20:38:48.904 INFO:teuthology.orchestra.run.smithi149.stdout:[global] 2023-09-18T20:38:48.904 INFO:teuthology.orchestra.run.smithi149.stdout: chdir = "" 2023-09-18T20:38:48.905 INFO:teuthology.orchestra.run.smithi149.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-18T20:38:48.905 INFO:teuthology.orchestra.run.smithi149.stdout: auth supported = cephx 2023-09-18T20:38:48.905 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.905 INFO:teuthology.orchestra.run.smithi149.stdout: filestore xattr use omap = true 2023-09-18T20:38:48.905 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.906 INFO:teuthology.orchestra.run.smithi149.stdout: mon clock drift allowed = 1.000 2023-09-18T20:38:48.906 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.906 INFO:teuthology.orchestra.run.smithi149.stdout: osd crush chooseleaf type = 0 2023-09-18T20:38:48.906 INFO:teuthology.orchestra.run.smithi149.stdout: auth debug = true 2023-09-18T20:38:48.907 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.907 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on old message = true 2023-09-18T20:38:48.907 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on bug = true 2023-09-18T20:38:48.907 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.907 INFO:teuthology.orchestra.run.smithi149.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-18T20:38:48.908 INFO:teuthology.orchestra.run.smithi149.stdout: mon pg warn max object skew = 0 2023-09-18T20:38:48.908 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.908 INFO:teuthology.orchestra.run.smithi149.stdout: # disable pg_autoscaler by default for new pools 2023-09-18T20:38:48.908 INFO:teuthology.orchestra.run.smithi149.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-18T20:38:48.909 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.909 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default size = 2 2023-09-18T20:38:48.909 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.909 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow primary affinity = true 2023-09-18T20:38:48.909 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow pg remap = true 2023-09-18T20:38:48.910 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on legacy crush tunables = false 2023-09-18T20:38:48.910 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on crush straw calc version zero = false 2023-09-18T20:38:48.910 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on no sortbitwise = false 2023-09-18T20:38:48.910 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on osd down out interval zero = false 2023-09-18T20:38:48.911 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on too few osds = false 2023-09-18T20:38:48.911 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-18T20:38:48.911 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-18T20:38:48.911 INFO:teuthology.orchestra.run.smithi149.stdout: mon_allow_pool_size_one = true 2023-09-18T20:38:48.911 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.912 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-18T20:38:48.912 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.912 INFO:teuthology.orchestra.run.smithi149.stdout: osd default data pool replay window = 5 2023-09-18T20:38:48.912 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.912 INFO:teuthology.orchestra.run.smithi149.stdout: mon allow pool delete = true 2023-09-18T20:38:48.913 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.913 INFO:teuthology.orchestra.run.smithi149.stdout: mon cluster log file level = debug 2023-09-18T20:38:48.913 INFO:teuthology.orchestra.run.smithi149.stdout: debug asserts on shutdown = true 2023-09-18T20:38:48.913 INFO:teuthology.orchestra.run.smithi149.stdout: mon health detail to clog = false 2023-09-18T20:38:48.914 INFO:teuthology.orchestra.run.smithi149.stdout: mon host = 172.21.15.149 2023-09-18T20:38:48.914 INFO:teuthology.orchestra.run.smithi149.stdout: mon client directed command retry = 5 2023-09-18T20:38:48.914 INFO:teuthology.orchestra.run.smithi149.stdout: ms inject socket failures = 5000 2023-09-18T20:38:48.914 INFO:teuthology.orchestra.run.smithi149.stdout: fsid = 6b7573aa-3580-401f-a189-159cf58e6075 2023-09-18T20:38:48.914 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.915 INFO:teuthology.orchestra.run.smithi149.stdout:[osd] 2023-09-18T20:38:48.915 INFO:teuthology.orchestra.run.smithi149.stdout: osd journal size = 100 2023-09-18T20:38:48.915 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.915 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub load threshold = 5.0 2023-09-18T20:38:48.916 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub max interval = 600 2023-09-18T20:38:48.916 INFO:teuthology.orchestra.run.smithi149.stdout: osd mclock profile = high_recovery_ops 2023-09-18T20:38:48.916 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.916 INFO:teuthology.orchestra.run.smithi149.stdout: osd recover clone overlap = true 2023-09-18T20:38:48.916 INFO:teuthology.orchestra.run.smithi149.stdout: osd recovery max chunk = 1048576 2023-09-18T20:38:48.917 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.917 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug shutdown = true 2023-09-18T20:38:48.917 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug op order = true 2023-09-18T20:38:48.917 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify stray on activate = true 2023-09-18T20:38:48.918 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.918 INFO:teuthology.orchestra.run.smithi149.stdout: osd open classes on start = true 2023-09-18T20:38:48.918 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug pg log writeout = true 2023-09-18T20:38:48.918 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.918 INFO:teuthology.orchestra.run.smithi149.stdout: osd deep scrub update digest min age = 30 2023-09-18T20:38:48.919 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.919 INFO:teuthology.orchestra.run.smithi149.stdout: osd map max advance = 10 2023-09-18T20:38:48.919 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.919 INFO:teuthology.orchestra.run.smithi149.stdout: journal zero on create = true 2023-09-18T20:38:48.920 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.920 INFO:teuthology.orchestra.run.smithi149.stdout: filestore ondisk finisher threads = 3 2023-09-18T20:38:48.920 INFO:teuthology.orchestra.run.smithi149.stdout: filestore apply finisher threads = 3 2023-09-18T20:38:48.920 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.920 INFO:teuthology.orchestra.run.smithi149.stdout: bdev debug aio = true 2023-09-18T20:38:48.921 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug misdirected ops = true 2023-09-18T20:38:48.921 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore block size = 96636764160 2023-09-18T20:38:48.921 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore compression algorithm = lz4 2023-09-18T20:38:48.921 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore compression mode = aggressive 2023-09-18T20:38:48.922 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore fsck on mount = True 2023-09-18T20:38:48.922 INFO:teuthology.orchestra.run.smithi149.stdout: debug bluefs = 1/20 2023-09-18T20:38:48.922 INFO:teuthology.orchestra.run.smithi149.stdout: debug bluestore = 1/20 2023-09-18T20:38:48.922 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-09-18T20:38:48.923 INFO:teuthology.orchestra.run.smithi149.stdout: debug osd = 20 2023-09-18T20:38:48.923 INFO:teuthology.orchestra.run.smithi149.stdout: debug rocksdb = 4/10 2023-09-18T20:38:48.923 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd backfillfull_ratio = 0.85 2023-09-18T20:38:48.923 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd full ratio = 0.9 2023-09-18T20:38:48.923 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd nearfull ratio = 0.8 2023-09-18T20:38:48.924 INFO:teuthology.orchestra.run.smithi149.stdout: osd failsafe full ratio = 0.95 2023-09-18T20:38:48.924 INFO:teuthology.orchestra.run.smithi149.stdout: osd objectstore = bluestore 2023-09-18T20:38:48.924 INFO:teuthology.orchestra.run.smithi149.stdout: osd shutdown pgref assert = True 2023-09-18T20:38:48.924 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.925 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr] 2023-09-18T20:38:48.925 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-09-18T20:38:48.925 INFO:teuthology.orchestra.run.smithi149.stdout: debug mgr = 20 2023-09-18T20:38:48.925 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2023-09-18T20:38:48.926 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2023-09-18T20:38:48.926 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2023-09-18T20:38:48.926 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2023-09-18T20:38:48.926 INFO:teuthology.orchestra.run.smithi149.stdout: mgr/telemetry/nag = false 2023-09-18T20:38:48.926 INFO:teuthology.orchestra.run.smithi149.stdout: debug_rbd = 20 2023-09-18T20:38:48.927 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.927 INFO:teuthology.orchestra.run.smithi149.stdout:[mon] 2023-09-18T20:38:48.927 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-09-18T20:38:48.927 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2023-09-18T20:38:48.928 INFO:teuthology.orchestra.run.smithi149.stdout: debug paxos = 20 2023-09-18T20:38:48.928 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2023-09-18T20:38:48.928 INFO:teuthology.orchestra.run.smithi149.stdout: mon data avail warn = 5 2023-09-18T20:38:48.928 INFO:teuthology.orchestra.run.smithi149.stdout: mon mgr mkfs grace = 240 2023-09-18T20:38:48.928 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2023-09-18T20:38:48.929 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd reporter subtree level = osd 2023-09-18T20:38:48.929 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd prime pg temp = true 2023-09-18T20:38:48.929 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2023-09-18T20:38:48.929 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.929 INFO:teuthology.orchestra.run.smithi149.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-18T20:38:48.930 INFO:teuthology.orchestra.run.smithi149.stdout: auth mon ticket ttl = 660 # 11m 2023-09-18T20:38:48.930 INFO:teuthology.orchestra.run.smithi149.stdout: auth service ticket ttl = 240 # 4m 2023-09-18T20:38:48.930 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.930 INFO:teuthology.orchestra.run.smithi149.stdout: # don't complain about insecure global_id in the test suite 2023-09-18T20:38:48.931 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-18T20:38:48.931 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-18T20:38:48.931 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.931 INFO:teuthology.orchestra.run.smithi149.stdout: # 1m isn't quite enough 2023-09-18T20:38:48.931 INFO:teuthology.orchestra.run.smithi149.stdout: mon_down_mkfs_grace = 2m 2023-09-18T20:38:48.932 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.932 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_filestore_osds = false 2023-09-18T20:38:48.932 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:38:48.932 INFO:teuthology.orchestra.run.smithi149.stdout:[client] 2023-09-18T20:38:48.933 INFO:teuthology.orchestra.run.smithi149.stdout: rgw cache enabled = true 2023-09-18T20:38:48.933 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable ops log = true 2023-09-18T20:38:48.933 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable usage log = true 2023-09-18T20:38:48.933 INFO:teuthology.orchestra.run.smithi149.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-18T20:38:48.933 INFO:teuthology.orchestra.run.smithi149.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-18T20:38:48.934 INFO:teuthology.orchestra.run.smithi149.stdout: rbd default features = 1 2023-09-18T20:38:48.934 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.a] 2023-09-18T20:38:48.937 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-18T20:38:48.937 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-18T20:38:49.034 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-18T20:38:49.035 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:49.035 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-18T20:38:49.051 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:49.052 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-18T20:38:49.108 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:38:49.109 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:49.109 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-18T20:38:49.109 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-18T20:38:49.185 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:38:49.186 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:38:49.242 INFO:tasks.ceph:Setting up mon nodes... 2023-09-18T20:38:49.243 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-18T20:38:49.243 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-18T20:38:49.336 INFO:teuthology.orchestra.run.smithi149.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-18T20:38:49.338 INFO:tasks.ceph:Setting up mds nodes... 2023-09-18T20:38:49.339 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-18T20:38:49.339 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-18T20:38:49.382 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-18T20:38:49.395 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-18T20:38:49.395 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-18T20:38:49.396 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-18T20:38:49.461 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-18T20:38:49.461 INFO:tasks.ceph:role: osd.0 2023-09-18T20:38:49.462 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:38:49.462 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-18T20:38:49.534 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-18T20:38:49.534 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-18T20:38:49.534 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-18T20:38:49.535 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-18T20:38:49.535 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-18T20:38:49.535 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-09-18T20:38:49.535 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-18T20:38:49.536 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-18T20:38:49.536 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-18T20:38:49.536 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-18T20:38:49.967 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-09-18T20:38:49.968 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-09-18T20:38:49.969 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-18T20:38:50.146 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-18T20:38:50.179 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-18T20:38:50.246 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-18T20:38:50.247 INFO:tasks.ceph:role: osd.1 2023-09-18T20:38:50.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:38:50.247 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-18T20:38:50.321 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-18T20:38:50.321 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-18T20:38:50.321 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-18T20:38:50.321 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-18T20:38:50.321 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-18T20:38:50.322 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-09-18T20:38:50.322 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-18T20:38:50.322 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-18T20:38:50.322 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-18T20:38:50.323 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-18T20:38:50.752 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-09-18T20:38:50.754 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-09-18T20:38:50.755 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-18T20:38:50.865 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-18T20:38:50.896 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-18T20:38:50.963 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-18T20:38:50.963 INFO:tasks.ceph:role: osd.2 2023-09-18T20:38:50.964 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:38:50.964 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-18T20:38:51.037 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-18T20:38:51.037 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-18T20:38:51.037 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-18T20:38:51.037 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-18T20:38:51.038 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-18T20:38:51.038 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-09-18T20:38:51.038 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-18T20:38:51.038 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-18T20:38:51.038 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-18T20:38:51.039 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-18T20:38:51.484 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-09-18T20:38:51.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-09-18T20:38:51.487 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-18T20:38:51.645 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-18T20:38:51.678 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:38:51.780 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:51.780+0000 7f5760bb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-18T20:38:51.781 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:51.781+0000 7f5760bb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-18T20:38:51.781 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:51.781+0000 7f5760bb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-18T20:38:51.781 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:51.781+0000 7f5760bb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-18T20:38:51.781 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:51.781+0000 7f5760bb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-18T20:38:51.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:51.789+0000 7f5760bb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-18T20:38:54.653 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-18T20:38:54.696 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:38:54.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:54.785+0000 7f97dfcb6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-18T20:38:54.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:54.785+0000 7f97dfcb6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-18T20:38:54.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:54.785+0000 7f97dfcb6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-18T20:38:54.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:54.785+0000 7f97dfcb6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-18T20:38:54.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:54.785+0000 7f97dfcb6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-18T20:38:54.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:54.785+0000 7f97dfcb6640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-18T20:38:57.655 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-18T20:38:57.686 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:38:57.776 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:57.775+0000 7fb08f3b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-18T20:38:57.776 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:57.775+0000 7fb08f3b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-18T20:38:57.776 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:57.775+0000 7fb08f3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-18T20:38:57.776 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:57.775+0000 7fb08f3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-18T20:38:57.777 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:57.775+0000 7fb08f3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-18T20:38:57.777 INFO:teuthology.orchestra.run.smithi149.stderr:2023-09-18T20:38:57.776+0000 7fb08f3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-18T20:39:00.632 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-18T20:39:00.662 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-18T20:39:00.662 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:00.663 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-18T20:39:00.729 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:00.730 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-18T20:39:00.797 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:00.798 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-18T20:39:00.864 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:00.865 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-18T20:39:00.931 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:00.932 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-18T20:39:00.988 INFO:tasks.ceph:Adding keys to all mons... 2023-09-18T20:39:00.989 DEBUG:teuthology.orchestra.run.smithi149:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-18T20:39:01.054 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr.x] 2023-09-18T20:39:01.054 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBZtQhlx0gZFBAATBohfTSLMmNOd2Y+b/zVzA== 2023-09-18T20:39:01.055 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.0] 2023-09-18T20:39:01.055 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBbtQhlGD+MLhAA2HwDGQEycm2Bx8SHAwTwKg== 2023-09-18T20:39:01.055 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.1] 2023-09-18T20:39:01.055 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBetQhlX3rWLhAAeAQANbAwoxvv0GUcu82tgg== 2023-09-18T20:39:01.055 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.2] 2023-09-18T20:39:01.056 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBhtQhlKqxELhAAcNqJuzq2lse7Aw5ez3GkIg== 2023-09-18T20:39:01.056 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2023-09-18T20:39:01.056 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBZtQhl69jaFhAAXYeFs21PlJJI1vszkx3R5Q== 2023-09-18T20:39:01.057 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-18T20:39:01.152 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-18T20:39:01.206 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-18T20:39:01.259 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-18T20:39:01.312 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-18T20:39:01.369 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-18T20:39:01.369 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-18T20:39:01.395 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-18T20:39:01.494 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-18T20:39:01.520 DEBUG:teuthology.orchestra.run.smithi149:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-18T20:39:01.575 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-18T20:39:01.576 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-18T20:39:01.576 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-18T20:39:01.618 INFO:tasks.ceph.mon.a:Started 2023-09-18T20:39:01.618 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-18T20:39:01.619 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-18T20:39:01.620 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-18T20:39:01.622 INFO:tasks.ceph.mgr.x:Started 2023-09-18T20:39:01.623 INFO:tasks.ceph:Setting crush tunables to default 2023-09-18T20:39:01.623 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd crush tunables default 2023-09-18T20:39:01.832 INFO:teuthology.orchestra.run.smithi149.stderr:adjusted tunables profile to default 2023-09-18T20:39:01.843 INFO:tasks.ceph:check_enable_crimson: False 2023-09-18T20:39:01.843 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-18T20:39:01.844 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:01.844 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-18T20:39:01.870 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:01.871 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-18T20:39:01.936 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:01.937 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-18T20:39:02.002 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 2738e12b-7bac-4194-8b8f-86cacbd2308f 0 2023-09-18T20:39:02.016 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.016+0000 7fc2dedc9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-18T20:39:02.088 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.088+0000 7fc2dedc9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-18T20:39:02.169 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.169+0000 7fc2dedc9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-18T20:39:02.230 INFO:teuthology.orchestra.run.smithi149.stdout:0 2023-09-18T20:39:02.243 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new e69b711f-10f7-4911-a025-cab999fb19b7 1 2023-09-18T20:39:02.251 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.252+0000 7fc2dedc9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-18T20:39:02.405 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.405+0000 7fc2dedc9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-18T20:39:02.457 INFO:teuthology.orchestra.run.smithi149.stdout:1 2023-09-18T20:39:02.471 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 978dafe6-0507-4167-89fe-b27a8496f2a5 2 2023-09-18T20:39:02.738 INFO:teuthology.orchestra.run.smithi149.stdout:2 2023-09-18T20:39:02.751 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-18T20:39:02.751 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-18T20:39:02.753 INFO:tasks.ceph.osd.0:Started 2023-09-18T20:39:02.754 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-18T20:39:02.754 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-18T20:39:02.756 INFO:tasks.ceph.osd.1:Started 2023-09-18T20:39:02.756 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-18T20:39:02.757 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-18T20:39:02.759 INFO:tasks.ceph.osd.2:Started 2023-09-18T20:39:02.759 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-18T20:39:02.841 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.841+0000 7fc2dedc9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-18T20:39:02.968 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:02.969+0000 7fc2dedc9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-18T20:39:02.969 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:02.969 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":5,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:39:02.736890+0000","last_up_change":"0.000000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:39:02.981 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-18T20:39:02.982 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-18T20:39:03.057 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:03.058+0000 7fc2dedc9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-18T20:39:03.140 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:03.140+0000 7fc2dedc9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-18T20:39:03.284 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-18T20:39:03.379 INFO:tasks.ceph.osd.0.smithi149.stderr:2023-09-18T20:39:03.380+0000 7fd310c0d640 -1 Falling back to public interface 2023-09-18T20:39:03.391 INFO:tasks.ceph.osd.2.smithi149.stderr:2023-09-18T20:39:03.391+0000 7f4b6ff12640 -1 Falling back to public interface 2023-09-18T20:39:03.392 INFO:tasks.ceph.osd.1.smithi149.stderr:2023-09-18T20:39:03.391+0000 7f78e3ae9640 -1 Falling back to public interface 2023-09-18T20:39:03.478 INFO:teuthology.misc.health.smithi149.stdout: 2023-09-18T20:39:03.478 INFO:teuthology.misc.health.smithi149.stdout:{"epoch":5,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:39:02.736890+0000","last_up_change":"0.000000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:39:03.489 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-18T20:39:03.596 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:03.596+0000 7fc2dedc9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-18T20:39:04.235 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:04.235+0000 7fc2dedc9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-18T20:39:04.305 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:04.306+0000 7fc2dedc9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-18T20:39:05.001 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:05.001+0000 7fc2dedc9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-18T20:39:05.080 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:05.080+0000 7fc2dedc9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-18T20:39:05.221 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:05.221+0000 7fc2dedc9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-18T20:39:05.460 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:05.460+0000 7fc2dedc9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-18T20:39:05.678 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:05.679+0000 7fc2dedc9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-18T20:39:05.811 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:05.811+0000 7fc2dedc9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-18T20:39:06.035 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:06.035+0000 7fc2dedc9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-18T20:39:06.227 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:06.227+0000 7fc2dedc9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-18T20:39:06.389 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:06.389+0000 7fc2dedc9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-18T20:39:06.722 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:06.722+0000 7fc2dedc9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-18T20:39:06.819 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:06.819+0000 7fc2dedc9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-18T20:39:06.880 INFO:tasks.ceph.osd.0.smithi149.stderr:2023-09-18T20:39:06.880+0000 7fd310c0d640 -1 osd.0 0 log_to_monitors true 2023-09-18T20:39:06.954 INFO:tasks.ceph.osd.2.smithi149.stderr:2023-09-18T20:39:06.954+0000 7f4b6ff12640 -1 osd.2 0 log_to_monitors true 2023-09-18T20:39:06.978 INFO:tasks.ceph.osd.1.smithi149.stderr:2023-09-18T20:39:06.979+0000 7f78e3ae9640 -1 osd.1 0 log_to_monitors true 2023-09-18T20:39:07.023 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:07.023+0000 7fc2dedc9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-18T20:39:07.619 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:07.619+0000 7fc2dedc9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-18T20:39:08.639 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:39:08.638+0000 7fc296d58700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-18T20:39:09.656 INFO:tasks.ceph.osd.2.smithi149.stderr:2023-09-18T20:39:09.655+0000 7f4b642ca700 -1 osd.2 0 waiting for initial osdmap 2023-09-18T20:39:09.671 INFO:tasks.ceph.osd.2.smithi149.stderr:2023-09-18T20:39:09.671+0000 7f4b5bea4700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-18T20:39:09.703 INFO:tasks.ceph.osd.0.smithi149.stderr:2023-09-18T20:39:09.702+0000 7fd304fc5700 -1 osd.0 0 waiting for initial osdmap 2023-09-18T20:39:09.709 INFO:tasks.ceph.osd.1.smithi149.stderr:2023-09-18T20:39:09.708+0000 7f78d7ea1700 -1 osd.1 0 waiting for initial osdmap 2023-09-18T20:39:09.719 INFO:tasks.ceph.osd.0.smithi149.stderr:2023-09-18T20:39:09.718+0000 7fd2fcb9f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-18T20:39:09.720 INFO:tasks.ceph.osd.1.smithi149.stderr:2023-09-18T20:39:09.720+0000 7f78cfa7b700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-18T20:39:09.792 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-18T20:39:10.103 INFO:teuthology.misc.health.smithi149.stdout: 2023-09-18T20:39:10.103 INFO:teuthology.misc.health.smithi149.stdout:{"epoch":7,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:39:08.709019+0000","last_up_change":"0.000000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:39:10.116 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-18T20:39:16.418 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-18T20:39:16.744 INFO:teuthology.misc.health.smithi149.stdout: 2023-09-18T20:39:16.744 INFO:teuthology.misc.health.smithi149.stdout:{"epoch":11,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:39:13.733661+0000","last_up_change":"2023-09-18T20:39:10.658176+0000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-18T20:39:11.677284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6801","nonce":1950803321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6803","nonce":1950803321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6807","nonce":1950803321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6805","nonce":1950803321}]},"public_addr":"172.21.15.149:6801/1950803321","cluster_addr":"172.21.15.149:6803/1950803321","heartbeat_back_addr":"172.21.15.149:6807/1950803321","heartbeat_front_addr":"172.21.15.149:6805/1950803321","state":["exists","up"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6812","nonce":1550375764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6817","nonce":1550375764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6823","nonce":1550375764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6821","nonce":1550375764}]},"public_addr":"172.21.15.149:6812/1550375764","cluster_addr":"172.21.15.149:6817/1550375764","heartbeat_back_addr":"172.21.15.149:6823/1550375764","heartbeat_front_addr":"172.21.15.149:6821/1550375764","state":["exists","up"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6809","nonce":2507427811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6813","nonce":2507427811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6819","nonce":2507427811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6816","nonce":2507427811}]},"public_addr":"172.21.15.149:6809/2507427811","cluster_addr":"172.21.15.149:6813/2507427811","heartbeat_back_addr":"172.21.15.149:6819/2507427811","heartbeat_front_addr":"172.21.15.149:6816/2507427811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.932185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.979717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.938873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:39:16.757 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-18T20:39:16.757 INFO:tasks.ceph:Creating RBD pool 2023-09-18T20:39:16.757 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-18T20:39:17.752 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'rbd' created 2023-09-18T20:39:17.764 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-18T20:39:18.755 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-18T20:39:18.767 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-18T20:39:18.767 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-18T20:39:18.768 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-18T20:39:18.768 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-18T20:39:18.769 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-18T20:39:19.147 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:19.160 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":1174285639},{"type":"v1","addr":"172.21.15.149:6825","nonce":1174285639}]},"active_addr":"172.21.15.149:6825/1174285639","active_change":"2023-09-18T20:39:07.632738+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":1715139781}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":3423697253}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":2882024060}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":3218437780}]}]} 2023-09-18T20:39:19.162 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-18T20:39:19.162 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-18T20:39:19.163 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-18T20:39:19.482 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:19.483 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":13,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:39:18.744756+0000","last_up_change":"2023-09-18T20:39:10.658176+0000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-18T20:39:11.677284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-18T20:39:17.094949+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6801","nonce":1950803321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6803","nonce":1950803321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6807","nonce":1950803321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6805","nonce":1950803321}]},"public_addr":"172.21.15.149:6801/1950803321","cluster_addr":"172.21.15.149:6803/1950803321","heartbeat_back_addr":"172.21.15.149:6807/1950803321","heartbeat_front_addr":"172.21.15.149:6805/1950803321","state":["exists","up"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6812","nonce":1550375764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6817","nonce":1550375764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6823","nonce":1550375764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6821","nonce":1550375764}]},"public_addr":"172.21.15.149:6812/1550375764","cluster_addr":"172.21.15.149:6817/1550375764","heartbeat_back_addr":"172.21.15.149:6823/1550375764","heartbeat_front_addr":"172.21.15.149:6821/1550375764","state":["exists","up"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6809","nonce":2507427811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6813","nonce":2507427811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6819","nonce":2507427811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6816","nonce":2507427811}]},"public_addr":"172.21.15.149:6809/2507427811","cluster_addr":"172.21.15.149:6813/2507427811","heartbeat_back_addr":"172.21.15.149:6819/2507427811","heartbeat_front_addr":"172.21.15.149:6816/2507427811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.932185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.979717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.938873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:39:19.496 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-18T20:39:19.497 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-18T20:39:19.817 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:19.817 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":14,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:39:19.749280+0000","last_up_change":"2023-09-18T20:39:10.658176+0000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-18T20:39:11.677284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-18T20:39:17.094949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6801","nonce":1950803321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6803","nonce":1950803321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6807","nonce":1950803321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6805","nonce":1950803321}]},"public_addr":"172.21.15.149:6801/1950803321","cluster_addr":"172.21.15.149:6803/1950803321","heartbeat_back_addr":"172.21.15.149:6807/1950803321","heartbeat_front_addr":"172.21.15.149:6805/1950803321","state":["exists","up"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6812","nonce":1550375764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6817","nonce":1550375764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6823","nonce":1550375764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6821","nonce":1550375764}]},"public_addr":"172.21.15.149:6812/1550375764","cluster_addr":"172.21.15.149:6817/1550375764","heartbeat_back_addr":"172.21.15.149:6823/1550375764","heartbeat_front_addr":"172.21.15.149:6821/1550375764","state":["exists","up"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6809","nonce":2507427811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6813","nonce":2507427811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6819","nonce":2507427811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6816","nonce":2507427811}]},"public_addr":"172.21.15.149:6809/2507427811","cluster_addr":"172.21.15.149:6813/2507427811","heartbeat_back_addr":"172.21.15.149:6819/2507427811","heartbeat_front_addr":"172.21.15.149:6816/2507427811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.932185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.979717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.938873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:39:19.831 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-18T20:39:19.832 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-18T20:39:19.832 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-18T20:39:20.002 INFO:teuthology.orchestra.run.smithi149.stdout:34359738372 2023-09-18T20:39:20.002 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-18T20:39:20.009 INFO:teuthology.orchestra.run.smithi149.stdout:34359738372 2023-09-18T20:39:20.009 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-18T20:39:20.010 INFO:teuthology.orchestra.run.smithi149.stdout:34359738372 2023-09-18T20:39:20.010 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-18T20:39:20.341 INFO:teuthology.orchestra.run.smithi149.stdout:34359738371 2023-09-18T20:39:20.355 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-09-18T20:39:20.387 INFO:teuthology.orchestra.run.smithi149.stdout:34359738371 2023-09-18T20:39:20.390 INFO:teuthology.orchestra.run.smithi149.stdout:34359738371 2023-09-18T20:39:20.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-09-18T20:39:20.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-09-18T20:39:21.357 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-18T20:39:21.405 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-18T20:39:21.407 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-18T20:39:21.709 INFO:teuthology.orchestra.run.smithi149.stdout:34359738371 2023-09-18T20:39:21.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-09-18T20:39:21.766 INFO:teuthology.orchestra.run.smithi149.stdout:34359738373 2023-09-18T20:39:21.773 INFO:teuthology.orchestra.run.smithi149.stdout:34359738373 2023-09-18T20:39:21.782 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.1 2023-09-18T20:39:21.783 DEBUG:teuthology.parallel:result is None 2023-09-18T20:39:21.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.2 2023-09-18T20:39:21.789 DEBUG:teuthology.parallel:result is None 2023-09-18T20:39:22.730 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-18T20:39:23.063 INFO:teuthology.orchestra.run.smithi149.stdout:34359738373 2023-09-18T20:39:23.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.0 2023-09-18T20:39:23.078 DEBUG:teuthology.parallel:result is None 2023-09-18T20:39:23.078 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-18T20:39:23.079 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:39:23.396 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:23.398 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:39:23.410 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-09-18T20:39:21.638693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81540,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033980,"statfs":{"total":289910292480,"available":289826795520,"internally_reserved":0,"allocated":475136,"data_stored":1281649,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000754"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883895+0000","last_change":"2023-09-18T20:39:18.754622+0000","last_active":"2023-09-18T20:39:19.883895+0000","last_peered":"2023-09-18T20:39:19.883895+0000","last_clean":"2023-09-18T20:39:19.883895+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:39:19.883895+0000","last_undegraded":"2023-09-18T20:39:19.883895+0000","last_fullsized":"2023-09-18T20:39:19.883895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:11:14.585598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.884034+0000","last_change":"2023-09-18T20:39:18.755105+0000","last_active":"2023-09-18T20:39:19.884034+0000","last_peered":"2023-09-18T20:39:19.884034+0000","last_clean":"2023-09-18T20:39:19.884034+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:39:19.884034+0000","last_undegraded":"2023-09-18T20:39:19.884034+0000","last_fullsized":"2023-09-18T20:39:19.884034+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:09:50.066257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883906+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:39:19.883906+0000","last_peered":"2023-09-18T20:39:19.883906+0000","last_clean":"2023-09-18T20:39:19.883906+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:39:19.883906+0000","last_undegraded":"2023-09-18T20:39:19.883906+0000","last_fullsized":"2023-09-18T20:39:19.883906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:15:46.444320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-18T20:39:18.755531+0000","last_change":"2023-09-18T20:39:18.753469+0000","last_active":"2023-09-18T20:39:18.755531+0000","last_peered":"2023-09-18T20:39:18.755531+0000","last_clean":"2023-09-18T20:39:18.755531+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:39:18.755531+0000","last_undegraded":"2023-09-18T20:39:18.755531+0000","last_fullsized":"2023-09-18T20:39:18.755531+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:29:46.154350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-18T20:39:18.747656+0000","last_change":"2023-09-18T20:39:12.868453+0000","last_active":"2023-09-18T20:39:18.747656+0000","last_peered":"2023-09-18T20:39:18.747656+0000","last_clean":"2023-09-18T20:39:18.747656+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:39:18.747656+0000","last_undegraded":"2023-09-18T20:39:18.747656+0000","last_fullsized":"2023-09-18T20:39:18.747656+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_clean_scrub_stamp":"2023-09-18T20:39:11.726727+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:32:10.120164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883837+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:39:19.883837+0000","last_peered":"2023-09-18T20:39:19.883837+0000","last_clean":"2023-09-18T20:39:19.883837+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:39:19.883837+0000","last_undegraded":"2023-09-18T20:39:19.883837+0000","last_fullsized":"2023-09-18T20:39:19.883837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:58:59.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-18T20:39:18.753984+0000","last_change":"2023-09-18T20:39:18.753462+0000","last_active":"2023-09-18T20:39:18.753984+0000","last_peered":"2023-09-18T20:39:18.753984+0000","last_clean":"2023-09-18T20:39:18.753984+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:39:18.753984+0000","last_undegraded":"2023-09-18T20:39:18.753984+0000","last_fullsized":"2023-09-18T20:39:18.753984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:06:02.461807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.752549+0000","last_change":"2023-09-18T20:39:18.753362+0000","last_active":"2023-09-18T20:39:19.752549+0000","last_peered":"2023-09-18T20:39:19.752549+0000","last_clean":"2023-09-18T20:39:19.752549+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:39:19.752549+0000","last_undegraded":"2023-09-18T20:39:19.752549+0000","last_fullsized":"2023-09-18T20:39:19.752549+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:57:37.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883784+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:39:19.883784+0000","last_peered":"2023-09-18T20:39:19.883784+0000","last_clean":"2023-09-18T20:39:19.883784+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:39:19.883784+0000","last_undegraded":"2023-09-18T20:39:19.883784+0000","last_fullsized":"2023-09-18T20:39:19.883784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:46:11.076182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":625165,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":625165,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:39:23.411 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:39:23.728 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:23.730 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:39:23.741 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-09-18T20:39:23.639000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81540,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033980,"statfs":{"total":289910292480,"available":289826795520,"internally_reserved":0,"allocated":475136,"data_stored":1281649,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001061"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883895+0000","last_change":"2023-09-18T20:39:18.754622+0000","last_active":"2023-09-18T20:39:19.883895+0000","last_peered":"2023-09-18T20:39:19.883895+0000","last_clean":"2023-09-18T20:39:19.883895+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:39:19.883895+0000","last_undegraded":"2023-09-18T20:39:19.883895+0000","last_fullsized":"2023-09-18T20:39:19.883895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:11:14.585598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.884034+0000","last_change":"2023-09-18T20:39:18.755105+0000","last_active":"2023-09-18T20:39:19.884034+0000","last_peered":"2023-09-18T20:39:19.884034+0000","last_clean":"2023-09-18T20:39:19.884034+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:39:19.884034+0000","last_undegraded":"2023-09-18T20:39:19.884034+0000","last_fullsized":"2023-09-18T20:39:19.884034+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:09:50.066257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883906+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:39:19.883906+0000","last_peered":"2023-09-18T20:39:19.883906+0000","last_clean":"2023-09-18T20:39:19.883906+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:39:19.883906+0000","last_undegraded":"2023-09-18T20:39:19.883906+0000","last_fullsized":"2023-09-18T20:39:19.883906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:15:46.444320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-18T20:39:18.755531+0000","last_change":"2023-09-18T20:39:18.753469+0000","last_active":"2023-09-18T20:39:18.755531+0000","last_peered":"2023-09-18T20:39:18.755531+0000","last_clean":"2023-09-18T20:39:18.755531+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:39:18.755531+0000","last_undegraded":"2023-09-18T20:39:18.755531+0000","last_fullsized":"2023-09-18T20:39:18.755531+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:29:46.154350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-18T20:39:18.747656+0000","last_change":"2023-09-18T20:39:12.868453+0000","last_active":"2023-09-18T20:39:18.747656+0000","last_peered":"2023-09-18T20:39:18.747656+0000","last_clean":"2023-09-18T20:39:18.747656+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:39:18.747656+0000","last_undegraded":"2023-09-18T20:39:18.747656+0000","last_fullsized":"2023-09-18T20:39:18.747656+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_clean_scrub_stamp":"2023-09-18T20:39:11.726727+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:32:10.120164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883837+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:39:19.883837+0000","last_peered":"2023-09-18T20:39:19.883837+0000","last_clean":"2023-09-18T20:39:19.883837+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:39:19.883837+0000","last_undegraded":"2023-09-18T20:39:19.883837+0000","last_fullsized":"2023-09-18T20:39:19.883837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:58:59.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-18T20:39:18.753984+0000","last_change":"2023-09-18T20:39:18.753462+0000","last_active":"2023-09-18T20:39:18.753984+0000","last_peered":"2023-09-18T20:39:18.753984+0000","last_clean":"2023-09-18T20:39:18.753984+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:39:18.753984+0000","last_undegraded":"2023-09-18T20:39:18.753984+0000","last_fullsized":"2023-09-18T20:39:18.753984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:06:02.461807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.752549+0000","last_change":"2023-09-18T20:39:18.753362+0000","last_active":"2023-09-18T20:39:19.752549+0000","last_peered":"2023-09-18T20:39:19.752549+0000","last_clean":"2023-09-18T20:39:19.752549+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:39:19.752549+0000","last_undegraded":"2023-09-18T20:39:19.752549+0000","last_fullsized":"2023-09-18T20:39:19.752549+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:57:37.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-18T20:39:19.883784+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:39:19.883784+0000","last_peered":"2023-09-18T20:39:19.883784+0000","last_clean":"2023-09-18T20:39:19.883784+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:39:19.883784+0000","last_undegraded":"2023-09-18T20:39:19.883784+0000","last_fullsized":"2023-09-18T20:39:19.883784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:46:11.076182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":625165,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":625165,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:39:23.743 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-18T20:39:23.743 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-18T20:39:23.743 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-18T20:39:23.744 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-18T20:39:24.107 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:24.107 INFO:teuthology.orchestra.run.smithi149.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-18T20:39:24.121 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-09-18T20:39:24.122 INFO:teuthology.run_tasks:Running task install... 2023-09-18T20:39:24.131 DEBUG:teuthology.task.install:project ceph 2023-09-18T20:39:24.131 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6'}} 2023-09-18T20:39:24.131 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6'} 2023-09-18T20:39:24.132 INFO:teuthology.task.install:Using flavor: default 2023-09-18T20:39:24.138 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-18T20:39:24.138 INFO:teuthology.task.install:extra packages: [] 2023-09-18T20:39:24.138 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['fio'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '277a824d757a39499f35213b338ab8a8372df0f6', 'tag': None, 'wait_for_package': False} 2023-09-18T20:39:24.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=277a824d757a39499f35213b338ab8a8372df0f6 2023-09-18T20:39:24.356 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/ 2023-09-18T20:39:24.356 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6170.g277a824d 2023-09-18T20:39:24.463 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-ajarr-repeated-rbd-client-blocklisting/277a824d757a39499f35213b338ab8a8372df0f6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-18T20:39:24.464 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:39:24.464 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-18T20:39:24.492 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, fio, fio on remote rpm x86_64 2023-09-18T20:39:24.492 DEBUG:teuthology.orchestra.run.smithi149:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/277a824d757a39499f35213b338ab8a8372df0f6/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-18T20:39:24.559 DEBUG:teuthology.orchestra.run.smithi149:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-18T20:39:24.626 DEBUG:teuthology.orchestra.run.smithi149:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-18T20:39:24.683 INFO:teuthology.orchestra.run.smithi149.stdout:check_obsoletes = 1 2023-09-18T20:39:24.694 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum clean all 2023-09-18T20:39:25.092 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:39:25.349 INFO:teuthology.orchestra.run.smithi149.stdout:53 files removed 2023-09-18T20:39:25.397 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd fio fio 2023-09-18T20:39:25.758 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-09-18T20:39:26.533 INFO:teuthology.orchestra.run.smithi149.stdout:ceph packages for x86_64 202 kB/s | 81 kB 00:00 2023-09-18T20:39:26.883 INFO:teuthology.orchestra.run.smithi149.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-09-18T20:39:27.125 INFO:teuthology.orchestra.run.smithi149.stdout:ceph source packages 6.8 kB/s | 1.5 kB 00:00 2023-09-18T20:39:27.293 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-09-18T20:39:28.086 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-09-18T20:39:31.620 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 465 kB/s | 24 kB 00:00 2023-09-18T20:39:32.521 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-09-18T20:39:40.814 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-09-18T20:39:51.835 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-radosgw-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.836 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-test-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.837 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.839 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-base-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.840 INFO:teuthology.orchestra.run.smithi149.stdout:Package cephadm-2:18.0.0-6170.g277a824d.el8.noarch is already installed. 2023-09-18T20:39:51.841 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-immutable-object-cache-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.842 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.843 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-dashboard-2:18.0.0-6170.g277a824d.el8.noarch is already installed. 2023-09-18T20:39:51.844 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-diskprediction-local-2:18.0.0-6170.g277a824d.el8.noarch is already installed. 2023-09-18T20:39:51.845 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-rook-2:18.0.0-6170.g277a824d.el8.noarch is already installed. 2023-09-18T20:39:51.847 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-cephadm-2:18.0.0-6170.g277a824d.el8.noarch is already installed. 2023-09-18T20:39:51.848 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.849 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-volume-2:18.0.0-6170.g277a824d.el8.noarch is already installed. 2023-09-18T20:39:51.850 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados-devel-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.851 INFO:teuthology.orchestra.run.smithi149.stdout:Package libcephfs2-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.852 INFO:teuthology.orchestra.run.smithi149.stdout:Package libcephfs-devel-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.854 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados2-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.855 INFO:teuthology.orchestra.run.smithi149.stdout:Package librbd1-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.856 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rados-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.857 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rgw-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.858 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-cephfs-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.859 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rbd-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.860 INFO:teuthology.orchestra.run.smithi149.stdout:Package rbd-fuse-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.862 INFO:teuthology.orchestra.run.smithi149.stdout:Package rbd-mirror-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.863 INFO:teuthology.orchestra.run.smithi149.stdout:Package rbd-nbd-2:18.0.0-6170.g277a824d.el8.x86_64 is already installed. 2023-09-18T20:39:51.974 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-09-18T20:39:51.975 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:39:51.975 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-09-18T20:39:51.975 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:39:51.975 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-09-18T20:39:51.976 INFO:teuthology.orchestra.run.smithi149.stdout: fio x86_64 3.19-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-18T20:39:51.976 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-09-18T20:39:51.976 INFO:teuthology.orchestra.run.smithi149.stdout: daxctl-libs x86_64 71.1-3.el8 rhel-8-for-x86_64-baseos-rpms 42 k 2023-09-18T20:39:51.976 INFO:teuthology.orchestra.run.smithi149.stdout: libpmemblk x86_64 1.6.1-1.el8 rhel-8-for-x86_64-appstream-rpms 92 k 2023-09-18T20:39:51.976 INFO:teuthology.orchestra.run.smithi149.stdout: ndctl-libs x86_64 71.1-3.el8 rhel-8-for-x86_64-baseos-rpms 79 k 2023-09-18T20:39:51.977 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:51.977 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-09-18T20:39:51.977 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-09-18T20:39:51.977 INFO:teuthology.orchestra.run.smithi149.stdout:Install 4 Packages 2023-09-18T20:39:51.977 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:51.978 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 750 k 2023-09-18T20:39:51.978 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 2.5 M 2023-09-18T20:39:51.979 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-09-18T20:39:52.058 INFO:teuthology.orchestra.run.smithi149.stdout:(1/4): daxctl-libs-71.1-3.el8.x86_64.rpm 514 kB/s | 42 kB 00:00 2023-09-18T20:39:52.083 INFO:teuthology.orchestra.run.smithi149.stdout:(2/4): ndctl-libs-71.1-3.el8.x86_64.rpm 740 kB/s | 79 kB 00:00 2023-09-18T20:39:52.108 INFO:teuthology.orchestra.run.smithi149.stdout:(3/4): libpmemblk-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 92 kB 00:00 2023-09-18T20:39:52.134 INFO:teuthology.orchestra.run.smithi149.stdout:(4/4): fio-3.19-3.el8.x86_64.rpm 3.4 MB/s | 538 kB 00:00 2023-09-18T20:39:52.134 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-09-18T20:39:52.134 INFO:teuthology.orchestra.run.smithi149.stdout:Total 4.6 MB/s | 750 kB 00:00 2023-09-18T20:39:52.176 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-09-18T20:39:52.188 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-09-18T20:39:52.189 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-09-18T20:39:52.263 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-09-18T20:39:52.267 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-09-18T20:39:52.473 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-09-18T20:39:52.616 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : daxctl-libs-71.1-3.el8.x86_64 1/4 2023-09-18T20:39:52.686 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: daxctl-libs-71.1-3.el8.x86_64 1/4 2023-09-18T20:39:52.874 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ndctl-libs-71.1-3.el8.x86_64 2/4 2023-09-18T20:39:52.947 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ndctl-libs-71.1-3.el8.x86_64 2/4 2023-09-18T20:39:53.108 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libpmemblk-1.6.1-1.el8.x86_64 3/4 2023-09-18T20:39:53.288 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libpmemblk-1.6.1-1.el8.x86_64 3/4 2023-09-18T20:39:53.480 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fio-3.19-3.el8.x86_64 4/4 2023-09-18T20:39:53.720 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: fio-3.19-3.el8.x86_64 4/4 2023-09-18T20:39:53.720 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : daxctl-libs-71.1-3.el8.x86_64 1/4 2023-09-18T20:39:53.720 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ndctl-libs-71.1-3.el8.x86_64 2/4 2023-09-18T20:39:53.721 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fio-3.19-3.el8.x86_64 3/4 2023-09-18T20:39:54.834 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libpmemblk-1.6.1-1.el8.x86_64 4/4 2023-09-18T20:39:54.834 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:14 ago on Mon 18 Sep 2023 08:39:40 PM UTC. 2023-09-18T20:39:57.218 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-09-18T20:39:57.402 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:57.403 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-09-18T20:39:57.403 INFO:teuthology.orchestra.run.smithi149.stdout: daxctl-libs-71.1-3.el8.x86_64 fio-3.19-3.el8.x86_64 2023-09-18T20:39:57.403 INFO:teuthology.orchestra.run.smithi149.stdout: libpmemblk-1.6.1-1.el8.x86_64 ndctl-libs-71.1-3.el8.x86_64 2023-09-18T20:39:57.403 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:39:57.403 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-09-18T20:39:57.560 DEBUG:teuthology.parallel:result is None 2023-09-18T20:39:57.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=277a824d757a39499f35213b338ab8a8372df0f6 2023-09-18T20:39:57.702 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-18T20:39:57.724 INFO:teuthology.orchestra.run.smithi149.stdout:18.0.0-6170.g277a824d.el8 2023-09-18T20:39:57.725 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6170.g277a824d.el8 2023-09-18T20:39:57.725 INFO:teuthology.task.install:The correct ceph version 18.0.0-6170.g277a824d is installed. 2023-09-18T20:39:57.725 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-09-18T20:39:57.726 INFO:teuthology.run_tasks:Running task workunit... 2023-09-18T20:39:57.737 INFO:tasks.workunit:Pulling workunits from ref daad63c9e55979797c86b6218776b5cf580dafc3 2023-09-18T20:39:57.738 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-18T20:39:57.738 DEBUG:teuthology.orchestra.run.smithi149:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-18T20:39:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-18T20:39:57.782 INFO:teuthology.orchestra.run.smithi149.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-09-18T20:39:57.782 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-09-18T20:39:57.838 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-09-18T20:39:57.838 DEBUG:teuthology.orchestra.run.smithi149:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-09-18T20:39:57.893 INFO:tasks.workunit:timeout=3h 2023-09-18T20:39:57.894 INFO:tasks.workunit:cleanup=True 2023-09-18T20:39:57.894 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ajarr/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout daad63c9e55979797c86b6218776b5cf580dafc3 2023-09-18T20:39:57.951 INFO:tasks.workunit.client.0.smithi149.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-18T20:40:35.955 INFO:tasks.workunit.client.0.smithi149.stderr:Note: switching to 'daad63c9e55979797c86b6218776b5cf580dafc3'. 2023-09-18T20:40:35.955 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.956 INFO:tasks.workunit.client.0.smithi149.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-18T20:40:35.956 INFO:tasks.workunit.client.0.smithi149.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-18T20:40:35.956 INFO:tasks.workunit.client.0.smithi149.stderr:state without impacting any branches by switching back to a branch. 2023-09-18T20:40:35.956 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.956 INFO:tasks.workunit.client.0.smithi149.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-18T20:40:35.957 INFO:tasks.workunit.client.0.smithi149.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-18T20:40:35.957 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.957 INFO:tasks.workunit.client.0.smithi149.stderr: git switch -c 2023-09-18T20:40:35.957 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.957 INFO:tasks.workunit.client.0.smithi149.stderr:Or undo this operation with: 2023-09-18T20:40:35.958 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.958 INFO:tasks.workunit.client.0.smithi149.stderr: git switch - 2023-09-18T20:40:35.958 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.958 INFO:tasks.workunit.client.0.smithi149.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-18T20:40:35.958 INFO:tasks.workunit.client.0.smithi149.stderr: 2023-09-18T20:40:35.959 INFO:tasks.workunit.client.0.smithi149.stderr:HEAD is now at daad63c9e55 qa/suites/rbd: add test to check rbd_support module recovery 2023-09-18T20:40:35.960 DEBUG:teuthology.orchestra.run.smithi149:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-18T20:40:36.026 INFO:tasks.workunit.client.0.smithi149.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-09-18T20:40:36.028 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-18T20:40:36.028 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-09-18T20:40:36.606 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-09-18T20:40:36.659 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-09-18T20:40:36.697 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-18T20:40:36.699 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-18T20:40:36.699 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-09-18T20:40:36.736 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-18T20:40:36.739 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-09-18T20:40:36.739 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-18T20:40:36.795 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-09-18T20:40:36.797 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-09-18T20:40:36.797 DEBUG:teuthology.orchestra.run.smithi149:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daad63c9e55979797c86b6218776b5cf580dafc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-09-18T20:40:36.856 INFO:tasks.workunit.client.0.smithi149.stderr:+ POOL=rbd2 2023-09-18T20:40:36.856 INFO:tasks.workunit.client.0.smithi149.stderr:+ IMAGE_PREFIX=image 2023-09-18T20:40:36.857 INFO:tasks.workunit.client.0.smithi149.stderr:+ MNT_DIR_PREFIX=mnt 2023-09-18T20:40:36.857 INFO:tasks.workunit.client.0.smithi149.stderr:+ IMAGES=5 2023-09-18T20:40:36.857 INFO:tasks.workunit.client.0.smithi149.stderr:+ RUN_TIME=180 2023-09-18T20:40:36.857 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd pool create rbd2 2023-09-18T20:40:37.903 INFO:tasks.workunit.client.0.smithi149.stderr:pool 'rbd2' already exists 2023-09-18T20:40:37.912 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd pool init rbd2 2023-09-18T20:40:40.794 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror pool enable rbd2 image 2023-09-18T20:40:40.838 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror pool peer add rbd2 cluster1 2023-09-18T20:40:40.874 INFO:tasks.workunit.client.0.smithi149.stdout:3b8deaf6-ee15-48d6-994c-be402bdca32a 2023-09-18T20:40:40.877 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i=1 )) 2023-09-18T20:40:40.877 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:40.877 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd create image1 --size 1024 --pool rbd2 2023-09-18T20:40:40.925 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror image enable rbd2/image1 snapshot 2023-09-18T20:40:41.813 INFO:tasks.workunit.client.0.smithi149.stdout:Mirroring enabled 2023-09-18T20:40:41.818 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 1m 2023-09-18T20:40:41.873 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:41.873 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:41.873 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd create image2 --size 1024 --pool rbd2 2023-09-18T20:40:41.917 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror image enable rbd2/image2 snapshot 2023-09-18T20:40:42.814 INFO:tasks.workunit.client.0.smithi149.stdout:Mirroring enabled 2023-09-18T20:40:42.820 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image2 1m 2023-09-18T20:40:42.872 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:42.873 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:42.873 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd create image3 --size 1024 --pool rbd2 2023-09-18T20:40:42.916 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror image enable rbd2/image3 snapshot 2023-09-18T20:40:43.823 INFO:tasks.workunit.client.0.smithi149.stdout:Mirroring enabled 2023-09-18T20:40:43.829 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image3 1m 2023-09-18T20:40:43.881 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:43.882 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:43.882 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd create image4 --size 1024 --pool rbd2 2023-09-18T20:40:43.927 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror image enable rbd2/image4 snapshot 2023-09-18T20:40:44.825 INFO:tasks.workunit.client.0.smithi149.stdout:Mirroring enabled 2023-09-18T20:40:44.830 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image4 1m 2023-09-18T20:40:44.882 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:44.883 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:44.883 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd create image5 --size 1024 --pool rbd2 2023-09-18T20:40:44.927 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror image enable rbd2/image5 snapshot 2023-09-18T20:40:45.826 INFO:tasks.workunit.client.0.smithi149.stdout:Mirroring enabled 2023-09-18T20:40:45.831 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image5 1m 2023-09-18T20:40:45.885 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:45.885 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:45.886 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i=1 )) 2023-09-18T20:40:45.886 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:45.886 INFO:tasks.workunit.client.0.smithi149.stderr:++ sudo rbd device map rbd2/image1 2023-09-18T20:40:46.050 INFO:tasks.workunit.client.0.smithi149.stderr:+ DEVS[$i]=/dev/rbd0 2023-09-18T20:40:46.051 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mkfs.ext4 /dev/rbd0 2023-09-18T20:40:46.102 INFO:tasks.workunit.client.0.smithi149.stderr:mke2fs 1.45.6 (20-Mar-2020) 2023-09-18T20:40:46.322 INFO:tasks.workunit.client.0.smithi149.stdout:Discarding device blocks: 4096/262144 done 2023-09-18T20:40:46.322 INFO:tasks.workunit.client.0.smithi149.stdout:Creating filesystem with 262144 4k blocks and 65536 inodes 2023-09-18T20:40:46.322 INFO:tasks.workunit.client.0.smithi149.stdout:Filesystem UUID: 4b3f0687-704e-493f-a68e-17af7aa4b4ca 2023-09-18T20:40:46.322 INFO:tasks.workunit.client.0.smithi149.stdout:Superblock backups stored on blocks: 2023-09-18T20:40:46.323 INFO:tasks.workunit.client.0.smithi149.stdout: 32768, 98304, 163840, 229376 2023-09-18T20:40:46.323 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:46.323 INFO:tasks.workunit.client.0.smithi149.stdout:Allocating group tables: 0/8 done 2023-09-18T20:40:46.324 INFO:tasks.workunit.client.0.smithi149.stdout:Writing inode tables: 0/8 done 2023-09-18T20:40:46.331 INFO:tasks.workunit.client.0.smithi149.stdout:Creating journal (8192 blocks): done 2023-09-18T20:40:46.339 INFO:tasks.workunit.client.0.smithi149.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2023-09-18T20:40:46.339 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:46.341 INFO:tasks.workunit.client.0.smithi149.stderr:+ MNTPATHS[$i]=mntimage1 2023-09-18T20:40:46.341 INFO:tasks.workunit.client.0.smithi149.stderr:+ mkdir mntimage1 2023-09-18T20:40:46.341 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mount /dev/rbd0 mntimage1 2023-09-18T20:40:46.374 INFO:tasks.workunit.client.0.smithi149.stderr:++ whoami 2023-09-18T20:40:46.375 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo chown ubuntu mntimage1 2023-09-18T20:40:46.386 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:46.386 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:46.386 INFO:tasks.workunit.client.0.smithi149.stderr:++ sudo rbd device map rbd2/image2 2023-09-18T20:40:46.445 INFO:tasks.workunit.client.0.smithi149.stderr:+ DEVS[$i]=/dev/rbd1 2023-09-18T20:40:46.445 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mkfs.ext4 /dev/rbd1 2023-09-18T20:40:46.455 INFO:tasks.workunit.client.0.smithi149.stderr:mke2fs 1.45.6 (20-Mar-2020) 2023-09-18T20:40:46.651 INFO:tasks.workunit.client.0.smithi149.stdout:Discarding device blocks: 4096/262144 done 2023-09-18T20:40:46.652 INFO:tasks.workunit.client.0.smithi149.stdout:Creating filesystem with 262144 4k blocks and 65536 inodes 2023-09-18T20:40:46.652 INFO:tasks.workunit.client.0.smithi149.stdout:Filesystem UUID: b608075d-995c-4dad-b8c8-26db9a49dd13 2023-09-18T20:40:46.652 INFO:tasks.workunit.client.0.smithi149.stdout:Superblock backups stored on blocks: 2023-09-18T20:40:46.652 INFO:tasks.workunit.client.0.smithi149.stdout: 32768, 98304, 163840, 229376 2023-09-18T20:40:46.653 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:46.653 INFO:tasks.workunit.client.0.smithi149.stdout:Allocating group tables: 0/8 done 2023-09-18T20:40:46.654 INFO:tasks.workunit.client.0.smithi149.stdout:Writing inode tables: 0/8 done 2023-09-18T20:40:46.662 INFO:tasks.workunit.client.0.smithi149.stdout:Creating journal (8192 blocks): done 2023-09-18T20:40:46.677 INFO:tasks.workunit.client.0.smithi149.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2023-09-18T20:40:46.677 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:46.679 INFO:tasks.workunit.client.0.smithi149.stderr:+ MNTPATHS[$i]=mntimage2 2023-09-18T20:40:46.679 INFO:tasks.workunit.client.0.smithi149.stderr:+ mkdir mntimage2 2023-09-18T20:40:46.679 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mount /dev/rbd1 mntimage2 2023-09-18T20:40:46.713 INFO:tasks.workunit.client.0.smithi149.stderr:++ whoami 2023-09-18T20:40:46.714 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo chown ubuntu mntimage2 2023-09-18T20:40:46.730 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:46.730 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:46.730 INFO:tasks.workunit.client.0.smithi149.stderr:++ sudo rbd device map rbd2/image3 2023-09-18T20:40:46.822 INFO:tasks.workunit.client.0.smithi149.stderr:+ DEVS[$i]=/dev/rbd2 2023-09-18T20:40:46.822 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mkfs.ext4 /dev/rbd2 2023-09-18T20:40:46.833 INFO:tasks.workunit.client.0.smithi149.stderr:mke2fs 1.45.6 (20-Mar-2020) 2023-09-18T20:40:47.038 INFO:tasks.workunit.client.0.smithi149.stdout:Discarding device blocks: 4096/262144 done 2023-09-18T20:40:47.038 INFO:tasks.workunit.client.0.smithi149.stdout:Creating filesystem with 262144 4k blocks and 65536 inodes 2023-09-18T20:40:47.038 INFO:tasks.workunit.client.0.smithi149.stdout:Filesystem UUID: 209127a4-5a21-4522-89d5-aa4989a0480a 2023-09-18T20:40:47.038 INFO:tasks.workunit.client.0.smithi149.stdout:Superblock backups stored on blocks: 2023-09-18T20:40:47.038 INFO:tasks.workunit.client.0.smithi149.stdout: 32768, 98304, 163840, 229376 2023-09-18T20:40:47.039 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:47.039 INFO:tasks.workunit.client.0.smithi149.stdout:Allocating group tables: 0/8 done 2023-09-18T20:40:47.042 INFO:tasks.workunit.client.0.smithi149.stdout:Writing inode tables: 0/8 done 2023-09-18T20:40:47.046 INFO:tasks.workunit.client.0.smithi149.stdout:Creating journal (8192 blocks): done 2023-09-18T20:40:47.056 INFO:tasks.workunit.client.0.smithi149.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2023-09-18T20:40:47.056 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:47.057 INFO:tasks.workunit.client.0.smithi149.stderr:+ MNTPATHS[$i]=mntimage3 2023-09-18T20:40:47.057 INFO:tasks.workunit.client.0.smithi149.stderr:+ mkdir mntimage3 2023-09-18T20:40:47.058 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mount /dev/rbd2 mntimage3 2023-09-18T20:40:47.091 INFO:tasks.workunit.client.0.smithi149.stderr:++ whoami 2023-09-18T20:40:47.092 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo chown ubuntu mntimage3 2023-09-18T20:40:47.108 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.108 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.108 INFO:tasks.workunit.client.0.smithi149.stderr:++ sudo rbd device map rbd2/image4 2023-09-18T20:40:47.182 INFO:tasks.workunit.client.0.smithi149.stderr:+ DEVS[$i]=/dev/rbd3 2023-09-18T20:40:47.182 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mkfs.ext4 /dev/rbd3 2023-09-18T20:40:47.192 INFO:tasks.workunit.client.0.smithi149.stderr:mke2fs 1.45.6 (20-Mar-2020) 2023-09-18T20:40:47.373 INFO:tasks.workunit.client.0.smithi149.stdout:Discarding device blocks: 4096/262144 done 2023-09-18T20:40:47.373 INFO:tasks.workunit.client.0.smithi149.stdout:Creating filesystem with 262144 4k blocks and 65536 inodes 2023-09-18T20:40:47.374 INFO:tasks.workunit.client.0.smithi149.stdout:Filesystem UUID: 1f52e000-3bf1-4829-a2c2-fa8d4fb4fcf2 2023-09-18T20:40:47.374 INFO:tasks.workunit.client.0.smithi149.stdout:Superblock backups stored on blocks: 2023-09-18T20:40:47.374 INFO:tasks.workunit.client.0.smithi149.stdout: 32768, 98304, 163840, 229376 2023-09-18T20:40:47.374 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:47.375 INFO:tasks.workunit.client.0.smithi149.stdout:Allocating group tables: 0/8 done 2023-09-18T20:40:47.378 INFO:tasks.workunit.client.0.smithi149.stdout:Writing inode tables: 0/8 done 2023-09-18T20:40:47.387 INFO:tasks.workunit.client.0.smithi149.stdout:Creating journal (8192 blocks): done 2023-09-18T20:40:47.401 INFO:tasks.workunit.client.0.smithi149.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2023-09-18T20:40:47.401 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:47.402 INFO:tasks.workunit.client.0.smithi149.stderr:+ MNTPATHS[$i]=mntimage4 2023-09-18T20:40:47.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ mkdir mntimage4 2023-09-18T20:40:47.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mount /dev/rbd3 mntimage4 2023-09-18T20:40:47.436 INFO:tasks.workunit.client.0.smithi149.stderr:++ whoami 2023-09-18T20:40:47.437 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo chown ubuntu mntimage4 2023-09-18T20:40:47.452 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.452 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.452 INFO:tasks.workunit.client.0.smithi149.stderr:++ sudo rbd device map rbd2/image5 2023-09-18T20:40:47.514 INFO:tasks.workunit.client.0.smithi149.stderr:+ DEVS[$i]=/dev/rbd4 2023-09-18T20:40:47.514 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mkfs.ext4 /dev/rbd4 2023-09-18T20:40:47.527 INFO:tasks.workunit.client.0.smithi149.stderr:mke2fs 1.45.6 (20-Mar-2020) 2023-09-18T20:40:47.704 INFO:tasks.workunit.client.0.smithi149.stdout:Discarding device blocks: 4096/262144 done 2023-09-18T20:40:47.704 INFO:tasks.workunit.client.0.smithi149.stdout:Creating filesystem with 262144 4k blocks and 65536 inodes 2023-09-18T20:40:47.704 INFO:tasks.workunit.client.0.smithi149.stdout:Filesystem UUID: 958c9f22-7db9-4d1c-b56f-0bb4a180689e 2023-09-18T20:40:47.704 INFO:tasks.workunit.client.0.smithi149.stdout:Superblock backups stored on blocks: 2023-09-18T20:40:47.705 INFO:tasks.workunit.client.0.smithi149.stdout: 32768, 98304, 163840, 229376 2023-09-18T20:40:47.705 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:47.705 INFO:tasks.workunit.client.0.smithi149.stdout:Allocating group tables: 0/8 done 2023-09-18T20:40:47.706 INFO:tasks.workunit.client.0.smithi149.stdout:Writing inode tables: 0/8 done 2023-09-18T20:40:47.708 INFO:tasks.workunit.client.0.smithi149.stdout:Creating journal (8192 blocks): done 2023-09-18T20:40:47.717 INFO:tasks.workunit.client.0.smithi149.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2023-09-18T20:40:47.717 INFO:tasks.workunit.client.0.smithi149.stdout: 2023-09-18T20:40:47.719 INFO:tasks.workunit.client.0.smithi149.stderr:+ MNTPATHS[$i]=mntimage5 2023-09-18T20:40:47.719 INFO:tasks.workunit.client.0.smithi149.stderr:+ mkdir mntimage5 2023-09-18T20:40:47.719 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo mount /dev/rbd4 mntimage5 2023-09-18T20:40:47.753 INFO:tasks.workunit.client.0.smithi149.stderr:++ whoami 2023-09-18T20:40:47.755 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo chown ubuntu mntimage5 2023-09-18T20:40:47.770 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.770 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.771 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i=1 )) 2023-09-18T20:40:47.771 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.771 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.772 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.772 INFO:tasks.workunit.client.0.smithi149.stderr:+ fio --name=fiotest --filename=mntimage1/test --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=180 --time_based 2023-09-18T20:40:47.772 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.772 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.772 INFO:tasks.workunit.client.0.smithi149.stderr:+ fio --name=fiotest --filename=mntimage2/test --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=180 --time_based 2023-09-18T20:40:47.773 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.774 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.775 INFO:tasks.workunit.client.0.smithi149.stderr:+ fio --name=fiotest --filename=mntimage3/test --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=180 --time_based 2023-09-18T20:40:47.775 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.775 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.775 INFO:tasks.workunit.client.0.smithi149.stderr:+ fio --name=fiotest --filename=mntimage4/test --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=180 --time_based 2023-09-18T20:40:47.775 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:40:47.775 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=5 )) 2023-09-18T20:40:47.776 INFO:tasks.workunit.client.0.smithi149.stderr:+ fio --name=fiotest --filename=mntimage5/test --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=180 --time_based 2023-09-18T20:40:47.776 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:40:47.776 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069647 2023-09-18T20:40:47.776 INFO:tasks.workunit.client.0.smithi149.stderr:+ END_TIME=1695069827 2023-09-18T20:40:47.776 INFO:tasks.workunit.client.0.smithi149.stderr:+ PREV_CLIENT=0 2023-09-18T20:40:47.776 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069647 -le 1695069827 ]] 2023-09-18T20:40:47.777 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:40:47.777 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:40:47.777 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:40:47.777 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:40:49.301 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2882024060 2023-09-18T20:40:49.301 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2882024060 ]] 2023-09-18T20:40:49.301 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2882024060 == \0 ]] 2023-09-18T20:40:49.301 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist add 172.21.15.149:0/2882024060 2023-09-18T20:40:51.789 INFO:tasks.workunit.client.0.smithi149.stderr:blocklisting 172.21.15.149:0/2882024060 until 2023-09-18T21:40:51.267934+0000 (3600 sec) 2023-09-18T20:40:51.813 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:41:01.815 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist ls 2023-09-18T20:41:01.815 INFO:tasks.workunit.client.0.smithi149.stderr:+ grep -q 172.21.15.149:0/2882024060 2023-09-18T20:41:02.564 INFO:tasks.workunit.client.0.smithi149.stderr:listed 1 entries 2023-09-18T20:41:02.579 INFO:tasks.workunit.client.0.smithi149.stderr:+ PREV_CLIENT=172.21.15.149:0/2882024060 2023-09-18T20:41:02.580 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:41:02.581 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069662 2023-09-18T20:41:02.581 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069662 -le 1695069827 ]] 2023-09-18T20:41:02.581 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:41:02.584 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:41:02.584 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:41:02.585 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:41:03.502 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2882024060 2023-09-18T20:41:03.502 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2882024060 ]] 2023-09-18T20:41:03.503 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2882024060 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\8\8\2\0\2\4\0\6\0 ]] 2023-09-18T20:41:03.503 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:41:13.504 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:41:13.506 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069673 2023-09-18T20:41:13.506 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069673 -le 1695069827 ]] 2023-09-18T20:41:13.506 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:41:13.507 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:41:13.507 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:41:13.507 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:41:14.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2882024060 2023-09-18T20:41:14.395 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2882024060 ]] 2023-09-18T20:41:14.395 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2882024060 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\8\8\2\0\2\4\0\6\0 ]] 2023-09-18T20:41:14.395 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:41:24.396 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:41:24.397 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069684 2023-09-18T20:41:24.397 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069684 -le 1695069827 ]] 2023-09-18T20:41:24.398 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:41:24.398 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:41:24.399 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:41:24.399 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:41:25.328 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/4070770118 2023-09-18T20:41:25.328 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/4070770118 ]] 2023-09-18T20:41:25.328 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/4070770118 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\8\8\2\0\2\4\0\6\0 ]] 2023-09-18T20:41:25.328 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist add 172.21.15.149:0/4070770118 2023-09-18T20:41:26.884 INFO:tasks.workunit.client.0.smithi149.stderr:blocklisting 172.21.15.149:0/4070770118 until 2023-09-18T21:41:26.110977+0000 (3600 sec) 2023-09-18T20:41:26.901 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:41:36.907 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist ls 2023-09-18T20:41:36.907 INFO:tasks.workunit.client.0.smithi149.stderr:+ grep -q 172.21.15.149:0/4070770118 2023-09-18T20:41:37.715 INFO:tasks.workunit.client.0.smithi149.stderr:listed 2 entries 2023-09-18T20:41:37.729 INFO:tasks.workunit.client.0.smithi149.stderr:+ PREV_CLIENT=172.21.15.149:0/4070770118 2023-09-18T20:41:37.729 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:41:37.730 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069697 2023-09-18T20:41:37.730 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069697 -le 1695069827 ]] 2023-09-18T20:41:37.738 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:41:37.739 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:41:37.739 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:41:37.739 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:41:38.678 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/4070770118 2023-09-18T20:41:38.678 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/4070770118 ]] 2023-09-18T20:41:38.678 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/4070770118 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\4\0\7\0\7\7\0\1\1\8 ]] 2023-09-18T20:41:38.679 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:41:48.679 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:41:48.680 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069708 2023-09-18T20:41:48.680 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069708 -le 1695069827 ]] 2023-09-18T20:41:48.681 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:41:48.683 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:41:48.683 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:41:48.691 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:41:49.631 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/4070770118 2023-09-18T20:41:49.632 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/4070770118 ]] 2023-09-18T20:41:49.632 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/4070770118 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\4\0\7\0\7\7\0\1\1\8 ]] 2023-09-18T20:41:49.632 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:41:59.634 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:41:59.635 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069719 2023-09-18T20:41:59.635 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069719 -le 1695069827 ]] 2023-09-18T20:41:59.637 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:41:59.637 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:41:59.637 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:41:59.637 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:42:00.010 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.009+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.011 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.009+0000 7fc268571700 -1 librbd::ImageState: 0x555946b56100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.011 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.009+0000 7fc267d70700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.011 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.010+0000 7fc268571700 -1 librbd::ImageState: 0x555946b56300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.011 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.011+0000 7fc267d70700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.012 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.011+0000 7fc267d70700 -1 librbd::ImageState: 0x555946261180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.016 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.016+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.016 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.016+0000 7fc268571700 -1 librbd::ImageState: 0x5559468f4680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.018 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.017+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.018 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:42:00.017+0000 7fc268571700 -1 librbd::ImageState: 0x555946b56000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:42:00.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/4070770118 2023-09-18T20:42:00.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/4070770118 ]] 2023-09-18T20:42:00.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/4070770118 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\4\0\7\0\7\7\0\1\1\8 ]] 2023-09-18T20:42:00.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:42:10.601 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:42:10.604 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069730 2023-09-18T20:42:10.604 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069730 -le 1695069827 ]] 2023-09-18T20:42:10.604 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:42:10.605 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:42:10.605 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:42:10.610 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:42:11.537 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/4070770118 2023-09-18T20:42:11.537 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/4070770118 ]] 2023-09-18T20:42:11.537 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/4070770118 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\4\0\7\0\7\7\0\1\1\8 ]] 2023-09-18T20:42:11.538 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:42:21.538 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:42:21.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069741 2023-09-18T20:42:21.540 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069741 -le 1695069827 ]] 2023-09-18T20:42:21.540 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:42:21.541 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:42:21.542 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:42:21.548 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:42:22.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/4070770118 2023-09-18T20:42:22.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/4070770118 ]] 2023-09-18T20:42:22.500 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/4070770118 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\4\0\7\0\7\7\0\1\1\8 ]] 2023-09-18T20:42:22.500 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:42:32.500 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:42:32.670 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069752 2023-09-18T20:42:32.670 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069752 -le 1695069827 ]] 2023-09-18T20:42:32.671 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:42:32.672 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:42:32.672 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:42:32.673 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:42:36.145 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2711493074 2023-09-18T20:42:36.146 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2711493074 ]] 2023-09-18T20:42:36.146 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2711493074 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\4\0\7\0\7\7\0\1\1\8 ]] 2023-09-18T20:42:36.146 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist add 172.21.15.149:0/2711493074 2023-09-18T20:42:38.935 INFO:tasks.workunit.client.0.smithi149.stderr:blocklisting 172.21.15.149:0/2711493074 until 2023-09-18T21:42:38.042717+0000 (3600 sec) 2023-09-18T20:42:38.952 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:42:48.954 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist ls 2023-09-18T20:42:48.954 INFO:tasks.workunit.client.0.smithi149.stderr:+ grep -q 172.21.15.149:0/2711493074 2023-09-18T20:42:50.250 INFO:tasks.workunit.client.0.smithi149.stderr:listed 3 entries 2023-09-18T20:42:50.266 INFO:tasks.workunit.client.0.smithi149.stderr:+ PREV_CLIENT=172.21.15.149:0/2711493074 2023-09-18T20:42:50.266 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:42:50.267 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069770 2023-09-18T20:42:50.267 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069770 -le 1695069827 ]] 2023-09-18T20:42:50.274 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:42:50.274 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:42:50.274 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:42:50.274 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:42:51.229 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2711493074 2023-09-18T20:42:51.229 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2711493074 ]] 2023-09-18T20:42:51.229 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2711493074 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\7\1\1\4\9\3\0\7\4 ]] 2023-09-18T20:42:51.229 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:43:00.010 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.005+0000 7fc268d72700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.010 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.005+0000 7fc268571700 -1 librbd::ImageState: 0x555946b56680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.010 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.008+0000 7fc268d72700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.011 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.008+0000 7fc268d72700 -1 librbd::ImageState: 0x555946b56880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.034 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.015+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.034 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.033+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.034 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.033+0000 7fc268571700 -1 librbd::ImageState: 0x555946b57d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.034 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.033+0000 7fc268571700 -1 librbd::ImageState: 0x555946b57f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.035 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.035+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:00.035 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:00.035+0000 7fc268571700 -1 librbd::ImageState: 0x555946ca2100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:01.230 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:43:01.231 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069781 2023-09-18T20:43:01.231 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069781 -le 1695069827 ]] 2023-09-18T20:43:01.232 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:43:01.232 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:43:01.232 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:43:01.233 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:43:02.903 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2711493074 2023-09-18T20:43:02.903 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2711493074 ]] 2023-09-18T20:43:02.903 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2711493074 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\7\1\1\4\9\3\0\7\4 ]] 2023-09-18T20:43:02.904 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:43:12.904 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:43:12.905 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069792 2023-09-18T20:43:12.905 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069792 -le 1695069827 ]] 2023-09-18T20:43:12.908 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:43:12.909 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:43:12.909 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:43:12.910 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:43:13.991 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/2711493074 2023-09-18T20:43:13.991 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/2711493074 ]] 2023-09-18T20:43:13.992 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/2711493074 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\7\1\1\4\9\3\0\7\4 ]] 2023-09-18T20:43:13.992 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:43:23.989 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:43:23.993 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069803 2023-09-18T20:43:23.993 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069803 -le 1695069827 ]] 2023-09-18T20:43:23.993 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:43:24.001 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:43:24.001 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:43:24.001 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:43:25.373 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/1940331705 2023-09-18T20:43:25.374 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/1940331705 ]] 2023-09-18T20:43:25.374 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/1940331705 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\2\7\1\1\4\9\3\0\7\4 ]] 2023-09-18T20:43:25.374 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist add 172.21.15.149:0/1940331705 2023-09-18T20:43:27.884 INFO:tasks.workunit.client.0.smithi149.stderr:blocklisting 172.21.15.149:0/1940331705 until 2023-09-18T21:43:27.107850+0000 (3600 sec) 2023-09-18T20:43:27.900 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:43:37.904 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph osd blocklist ls 2023-09-18T20:43:37.904 INFO:tasks.workunit.client.0.smithi149.stderr:+ grep -q 172.21.15.149:0/1940331705 2023-09-18T20:43:39.061 INFO:tasks.workunit.client.0.smithi149.stderr:listed 4 entries 2023-09-18T20:43:39.084 INFO:tasks.workunit.client.0.smithi149.stderr:+ PREV_CLIENT=172.21.15.149:0/1940331705 2023-09-18T20:43:39.085 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:43:39.085 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069819 2023-09-18T20:43:39.086 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069819 -le 1695069827 ]] 2023-09-18T20:43:39.086 INFO:tasks.workunit.client.0.smithi149.stderr:++ ceph mgr dump 2023-09-18T20:43:39.086 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq '.active_clients[]' 2023-09-18T20:43:39.087 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-18T20:43:39.090 INFO:tasks.workunit.client.0.smithi149.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-18T20:43:40.093 INFO:tasks.workunit.client.0.smithi149.stderr:+ CLIENT_ADDR=172.21.15.149:0/1940331705 2023-09-18T20:43:40.094 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ -z 172.21.15.149:0/1940331705 ]] 2023-09-18T20:43:40.094 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 172.21.15.149:0/1940331705 == \1\7\2\.\2\1\.\1\5\.\1\4\9\:\0\/\1\9\4\0\3\3\1\7\0\5 ]] 2023-09-18T20:43:40.094 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:43:50.097 INFO:tasks.workunit.client.0.smithi149.stderr:++ date +%s 2023-09-18T20:43:50.097 INFO:tasks.workunit.client.0.smithi149.stderr:+ CURRENT_TIME=1695069830 2023-09-18T20:43:50.097 INFO:tasks.workunit.client.0.smithi149.stderr:+ [[ 1695069830 -le 1695069827 ]] 2023-09-18T20:43:50.097 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i=1 )) 2023-09-18T20:43:50.097 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=24 )) 2023-09-18T20:43:50.098 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 2m 2023-09-18T20:43:50.245 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:50.242+0000 7fc268571700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:50.245 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:50.242+0000 7fc268571700 -1 librbd::ImageState: 0x5559468f5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-18T20:43:50.251 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:43:50.251+0000 7fc264569700 -1 mgr.server reply reply (11) Resource temporarily unavailable [errno 108] RBD connection was shutdown (error opening image b'image1' at snapshot None) 2023-09-18T20:43:50.251 INFO:tasks.workunit.client.0.smithi149.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: [errno 108] RBD connection was shutdown (error opening image b'image1' at snapshot None) 2023-09-18T20:43:50.259 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:44:00.263 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:44:00.264 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=24 )) 2023-09-18T20:44:00.264 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 2m 2023-09-18T20:44:00.304 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:44:00.304+0000 7fc264569700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-18T20:44:00.305 INFO:tasks.workunit.client.0.smithi149.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-18T20:44:00.307 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:44:10.309 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:44:10.309 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=24 )) 2023-09-18T20:44:10.309 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 2m 2023-09-18T20:44:10.350 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:44:10.350+0000 7fc264569700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-18T20:44:10.350 INFO:tasks.workunit.client.0.smithi149.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-18T20:44:10.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:44:20.354 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:44:20.355 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=24 )) 2023-09-18T20:44:20.355 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 2m 2023-09-18T20:44:20.396 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:44:20.396+0000 7fc264569700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-18T20:44:20.397 INFO:tasks.workunit.client.0.smithi149.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-18T20:44:20.399 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:44:30.401 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:44:30.401 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=24 )) 2023-09-18T20:44:30.401 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 2m 2023-09-18T20:44:30.441 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-09-18T20:44:30.440+0000 7fc264569700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-18T20:44:30.442 INFO:tasks.workunit.client.0.smithi149.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-18T20:44:30.444 INFO:tasks.workunit.client.0.smithi149.stderr:+ sleep 10 2023-09-18T20:44:40.446 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i++ )) 2023-09-18T20:44:40.446 INFO:tasks.workunit.client.0.smithi149.stderr:+ (( i<=24 )) 2023-09-18T20:44:40.446 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule add -p rbd2 --image image1 2m 2023-09-18T20:44:40.505 INFO:tasks.workunit.client.0.smithi149.stderr:+ break 2023-09-18T20:44:40.506 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule ls -p rbd2 --image image1 2023-09-18T20:44:40.506 INFO:tasks.workunit.client.0.smithi149.stderr:+ grep 'every 2m' 2023-09-18T20:44:40.554 INFO:tasks.workunit.client.0.smithi149.stdout:every 1m, every 2m 2023-09-18T20:44:40.555 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule ls -p rbd2 --image image1 2023-09-18T20:44:40.555 INFO:tasks.workunit.client.0.smithi149.stderr:+ grep 'every 1m' 2023-09-18T20:44:40.601 INFO:tasks.workunit.client.0.smithi149.stdout:every 1m, every 2m 2023-09-18T20:44:40.602 INFO:tasks.workunit.client.0.smithi149.stderr:+ rbd mirror snapshot schedule rm -p rbd2 --image image1 2m 2023-09-18T20:44:40.652 INFO:tasks.workunit.client.0.smithi149.stderr:+ sudo pkill fio 2023-09-18T20:44:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-18T20:44:40.747 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-09-18T20:44:40.747 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-18T20:44:41.061 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daad63c9e55979797c86b6218776b5cf580dafc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-18T20:44:41.133 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8936a200c4fa47779fa5c8d88d26e0a3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_daad63c9e55979797c86b6218776b5cf580dafc3/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daad63c9e55979797c86b6218776b5cf580dafc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-18T20:44:41.136 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-18T20:44:41.192 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-18T20:44:41.201 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-18T20:44:41.202 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:44:41.530 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:44:41.531 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:44:41.543 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-18T20:44:39.724104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216906620,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":169021,"num_read_kb":672741,"num_write":169186,"num_write_kb":4772713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171642,"ondisk_log_size":171642,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9619752,"kb_used_data":8196816,"kb_used_omap":4,"kb_used_meta":1422907,"kb_avail":273495768,"statfs":{"total":289910292480,"available":280059666432,"internally_reserved":0,"allocated":8393539584,"data_stored":8401751201,"data_compressed":333260,"data_compressed_allocated":2252800,"data_compressed_original":11051008,"omap_allocated":4782,"internal_metadata":1457057106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":208,"num_read_kb":180,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"3.1f","version":"31'5467","reported_seq":10804,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.490803+0000","last_change":"2023-09-18T20:40:40.833224+0000","last_active":"2023-09-18T20:43:57.490803+0000","last_peered":"2023-09-18T20:43:57.490803+0000","last_clean":"2023-09-18T20:43:57.490803+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:43:57.490803+0000","last_undegraded":"2023-09-18T20:43:57.490803+0000","last_fullsized":"2023-09-18T20:43:57.490803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:34:11.826626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4969","reported_seq":9779,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.488616+0000","last_change":"2023-09-18T20:40:40.786113+0000","last_active":"2023-09-18T20:43:57.488616+0000","last_peered":"2023-09-18T20:43:57.488616+0000","last_clean":"2023-09-18T20:43:57.488616+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:43:57.488616+0000","last_undegraded":"2023-09-18T20:43:57.488616+0000","last_fullsized":"2023-09-18T20:43:57.488616+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T20:55:47.617213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'5712","reported_seq":11417,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.449589+0000","last_change":"2023-09-18T20:40:40.785804+0000","last_active":"2023-09-18T20:43:57.449589+0000","last_peered":"2023-09-18T20:43:57.449589+0000","last_clean":"2023-09-18T20:43:57.449589+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:43:57.449589+0000","last_undegraded":"2023-09-18T20:43:57.449589+0000","last_fullsized":"2023-09-18T20:43:57.449589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:50:15.757762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614528,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5659,"num_write_kb":161724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5892","reported_seq":11756,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.200432+0000","last_change":"2023-09-18T20:40:40.790340+0000","last_active":"2023-09-18T20:44:33.200432+0000","last_peered":"2023-09-18T20:44:33.200432+0000","last_clean":"2023-09-18T20:44:33.200432+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:44:33.200432+0000","last_undegraded":"2023-09-18T20:44:33.200432+0000","last_fullsized":"2023-09-18T20:44:33.200432+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:25:56.139626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5838,"num_read_kb":23171,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5576","reported_seq":11077,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.480878+0000","last_change":"2023-09-18T20:40:40.832137+0000","last_active":"2023-09-18T20:43:57.480878+0000","last_peered":"2023-09-18T20:43:57.480878+0000","last_clean":"2023-09-18T20:43:57.480878+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:43:57.480878+0000","last_undegraded":"2023-09-18T20:43:57.480878+0000","last_fullsized":"2023-09-18T20:43:57.480878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:32:54.275405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"31'4802","reported_seq":9860,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.199077+0000","last_change":"2023-09-18T20:40:40.833200+0000","last_active":"2023-09-18T20:44:33.199077+0000","last_peered":"2023-09-18T20:44:33.199077+0000","last_clean":"2023-09-18T20:44:33.199077+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:44:33.199077+0000","last_undegraded":"2023-09-18T20:44:33.199077+0000","last_fullsized":"2023-09-18T20:44:33.199077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:06:48.959386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124866560,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5078,"num_read_kb":20357,"num_write":4767,"num_write_kb":138221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.888009+0000","last_change":"2023-09-18T20:39:18.754622+0000","last_active":"2023-09-18T20:43:27.888009+0000","last_peered":"2023-09-18T20:43:27.888009+0000","last_clean":"2023-09-18T20:43:27.888009+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:43:27.888009+0000","last_undegraded":"2023-09-18T20:43:27.888009+0000","last_fullsized":"2023-09-18T20:43:27.888009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:11:14.585598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887711+0000","last_change":"2023-09-18T20:39:18.755105+0000","last_active":"2023-09-18T20:43:27.887711+0000","last_peered":"2023-09-18T20:43:27.887711+0000","last_clean":"2023-09-18T20:43:27.887711+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:43:27.887711+0000","last_undegraded":"2023-09-18T20:43:27.887711+0000","last_fullsized":"2023-09-18T20:43:27.887711+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:09:50.066257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'4660","reported_seq":9362,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.431297+0000","last_change":"2023-09-18T20:40:40.786389+0000","last_active":"2023-09-18T20:43:57.431297+0000","last_peered":"2023-09-18T20:43:57.431297+0000","last_clean":"2023-09-18T20:43:57.431297+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:43:57.431297+0000","last_undegraded":"2023-09-18T20:43:57.431297+0000","last_fullsized":"2023-09-18T20:43:57.431297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:01:55.705787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997568,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4614,"num_write_kb":133028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'7540","reported_seq":15114,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.499606+0000","last_change":"2023-09-18T20:40:40.791026+0000","last_active":"2023-09-18T20:43:58.499606+0000","last_peered":"2023-09-18T20:43:58.499606+0000","last_clean":"2023-09-18T20:43:58.499606+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:43:58.499606+0000","last_undegraded":"2023-09-18T20:43:58.499606+0000","last_fullsized":"2023-09-18T20:43:58.499606+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:47:41.555603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.891231+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:43:27.891231+0000","last_peered":"2023-09-18T20:43:27.891231+0000","last_clean":"2023-09-18T20:43:27.891231+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:43:27.891231+0000","last_undegraded":"2023-09-18T20:43:27.891231+0000","last_fullsized":"2023-09-18T20:43:27.891231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:15:46.444320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6505","reported_seq":12883,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.464898+0000","last_change":"2023-09-18T20:40:40.831880+0000","last_active":"2023-09-18T20:43:57.464898+0000","last_peered":"2023-09-18T20:43:57.464898+0000","last_clean":"2023-09-18T20:43:57.464898+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:43:57.464898+0000","last_undegraded":"2023-09-18T20:43:57.464898+0000","last_fullsized":"2023-09-18T20:43:57.464898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:39:27.193036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.873392+0000","last_change":"2023-09-18T20:39:18.753469+0000","last_active":"2023-09-18T20:43:26.873392+0000","last_peered":"2023-09-18T20:43:26.873392+0000","last_clean":"2023-09-18T20:43:26.873392+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:43:26.873392+0000","last_undegraded":"2023-09-18T20:43:26.873392+0000","last_fullsized":"2023-09-18T20:43:26.873392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:22:56.897789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'4259","reported_seq":8454,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.452680+0000","last_change":"2023-09-18T20:40:40.790444+0000","last_active":"2023-09-18T20:43:57.452680+0000","last_peered":"2023-09-18T20:43:57.452680+0000","last_clean":"2023-09-18T20:43:57.452680+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:43:57.452680+0000","last_undegraded":"2023-09-18T20:43:57.452680+0000","last_fullsized":"2023-09-18T20:43:57.452680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:15:42.288917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":159,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.880206+0000","last_change":"2023-09-18T20:39:12.868453+0000","last_active":"2023-09-18T20:43:26.880206+0000","last_peered":"2023-09-18T20:43:26.880206+0000","last_clean":"2023-09-18T20:43:26.880206+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:43:26.880206+0000","last_undegraded":"2023-09-18T20:43:26.880206+0000","last_fullsized":"2023-09-18T20:43:26.880206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_clean_scrub_stamp":"2023-09-18T20:39:11.726727+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:32:10.120164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887509+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:43:27.887509+0000","last_peered":"2023-09-18T20:43:27.887509+0000","last_clean":"2023-09-18T20:43:27.887509+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:43:27.887509+0000","last_undegraded":"2023-09-18T20:43:27.887509+0000","last_fullsized":"2023-09-18T20:43:27.887509+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:58:59.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'5837","reported_seq":11562,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.492284+0000","last_change":"2023-09-18T20:40:40.790900+0000","last_active":"2023-09-18T20:43:58.492284+0000","last_peered":"2023-09-18T20:43:58.492284+0000","last_clean":"2023-09-18T20:43:58.492284+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:43:58.492284+0000","last_undegraded":"2023-09-18T20:43:58.492284+0000","last_fullsized":"2023-09-18T20:43:58.492284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:17:11.468556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.875264+0000","last_change":"2023-09-18T20:39:18.753462+0000","last_active":"2023-09-18T20:43:26.875264+0000","last_peered":"2023-09-18T20:43:26.875264+0000","last_clean":"2023-09-18T20:43:26.875264+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:43:26.875264+0000","last_undegraded":"2023-09-18T20:43:26.875264+0000","last_fullsized":"2023-09-18T20:43:26.875264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:13:27.835062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"31'5805","reported_seq":11502,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.500020+0000","last_change":"2023-09-18T20:40:40.832355+0000","last_active":"2023-09-18T20:43:57.500020+0000","last_peered":"2023-09-18T20:43:57.500020+0000","last_clean":"2023-09-18T20:43:57.500020+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:43:57.500020+0000","last_undegraded":"2023-09-18T20:43:57.500020+0000","last_fullsized":"2023-09-18T20:43:57.500020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:44:19.351290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.147408+0000","last_change":"2023-09-18T20:39:18.753362+0000","last_active":"2023-09-18T20:43:27.147408+0000","last_peered":"2023-09-18T20:43:27.147408+0000","last_clean":"2023-09-18T20:43:27.147408+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:43:27.147408+0000","last_undegraded":"2023-09-18T20:43:27.147408+0000","last_fullsized":"2023-09-18T20:43:27.147408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:57:37.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6170","reported_seq":12274,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.430133+0000","last_change":"2023-09-18T20:40:40.786258+0000","last_active":"2023-09-18T20:43:57.430133+0000","last_peered":"2023-09-18T20:43:57.430133+0000","last_clean":"2023-09-18T20:43:57.430133+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:43:57.430133+0000","last_undegraded":"2023-09-18T20:43:57.430133+0000","last_fullsized":"2023-09-18T20:43:57.430133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:33:56.555234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.889508+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:43:27.889508+0000","last_peered":"2023-09-18T20:43:27.889508+0000","last_clean":"2023-09-18T20:43:27.889508+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:43:27.889508+0000","last_undegraded":"2023-09-18T20:43:27.889508+0000","last_fullsized":"2023-09-18T20:43:27.889508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:46:11.076182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"31'5238","reported_seq":10420,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486825+0000","last_change":"2023-09-18T20:40:40.785694+0000","last_active":"2023-09-18T20:43:57.486825+0000","last_peered":"2023-09-18T20:43:57.486825+0000","last_clean":"2023-09-18T20:43:57.486825+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:43:57.486825+0000","last_undegraded":"2023-09-18T20:43:57.486825+0000","last_fullsized":"2023-09-18T20:43:57.486825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:48:25.383550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"31'3820","reported_seq":7550,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.197751+0000","last_change":"2023-09-18T20:40:40.831884+0000","last_active":"2023-09-18T20:44:33.197751+0000","last_peered":"2023-09-18T20:44:33.197751+0000","last_clean":"2023-09-18T20:44:33.197751+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:44:33.197751+0000","last_undegraded":"2023-09-18T20:44:33.197751+0000","last_fullsized":"2023-09-18T20:44:33.197751+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:26:57.735585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3828,"num_read_kb":14459,"num_write":3566,"num_write_kb":100156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"31'7146","reported_seq":13922,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.195586+0000","last_change":"2023-09-18T20:40:40.832373+0000","last_active":"2023-09-18T20:44:33.195586+0000","last_peered":"2023-09-18T20:44:33.195586+0000","last_clean":"2023-09-18T20:44:33.195586+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:44:33.195586+0000","last_undegraded":"2023-09-18T20:44:33.195586+0000","last_fullsized":"2023-09-18T20:44:33.195586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:23:31.794226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":6873,"num_read_kb":27082,"num_write":6864,"num_write_kb":191656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'3581","reported_seq":7123,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.402162+0000","last_change":"2023-09-18T20:40:40.790576+0000","last_active":"2023-09-18T20:43:57.402162+0000","last_peered":"2023-09-18T20:43:57.402162+0000","last_clean":"2023-09-18T20:43:57.402162+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:43:57.402162+0000","last_undegraded":"2023-09-18T20:43:57.402162+0000","last_fullsized":"2023-09-18T20:43:57.402162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:13:44.471080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'4630","reported_seq":9019,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.199745+0000","last_change":"2023-09-18T20:40:40.832593+0000","last_active":"2023-09-18T20:44:33.199745+0000","last_peered":"2023-09-18T20:44:33.199745+0000","last_clean":"2023-09-18T20:44:33.199745+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:44:33.199745+0000","last_undegraded":"2023-09-18T20:44:33.199745+0000","last_fullsized":"2023-09-18T20:44:33.199745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:21:52.199914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108081152,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4486,"num_read_kb":17534,"num_write":4368,"num_write_kb":120480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"31'4114","reported_seq":8338,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.454474+0000","last_change":"2023-09-18T20:40:40.791131+0000","last_active":"2023-09-18T20:43:57.454474+0000","last_peered":"2023-09-18T20:43:57.454474+0000","last_clean":"2023-09-18T20:43:57.454474+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:43:57.454474+0000","last_undegraded":"2023-09-18T20:43:57.454474+0000","last_fullsized":"2023-09-18T20:43:57.454474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:05:41.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"31'6084","reported_seq":12190,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.488321+0000","last_change":"2023-09-18T20:40:40.786504+0000","last_active":"2023-09-18T20:43:57.488321+0000","last_peered":"2023-09-18T20:43:57.488321+0000","last_clean":"2023-09-18T20:43:57.488321+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:43:57.488321+0000","last_undegraded":"2023-09-18T20:43:57.488321+0000","last_fullsized":"2023-09-18T20:43:57.488321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:22:28.068618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'4159","reported_seq":8163,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.196731+0000","last_change":"2023-09-18T20:40:40.833389+0000","last_active":"2023-09-18T20:44:33.196731+0000","last_peered":"2023-09-18T20:44:33.196731+0000","last_clean":"2023-09-18T20:44:33.196731+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:44:33.196731+0000","last_undegraded":"2023-09-18T20:44:33.196731+0000","last_fullsized":"2023-09-18T20:44:33.196731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:27.656176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4108,"num_read_kb":15537,"num_write":3893,"num_write_kb":109673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"31'4527","reported_seq":9042,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486241+0000","last_change":"2023-09-18T20:40:40.789756+0000","last_active":"2023-09-18T20:43:57.486241+0000","last_peered":"2023-09-18T20:43:57.486241+0000","last_clean":"2023-09-18T20:43:57.486241+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:43:57.486241+0000","last_undegraded":"2023-09-18T20:43:57.486241+0000","last_fullsized":"2023-09-18T20:43:57.486241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:33:32.442279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'6247","reported_seq":12323,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.495507+0000","last_change":"2023-09-18T20:40:40.785973+0000","last_active":"2023-09-18T20:43:58.495507+0000","last_peered":"2023-09-18T20:43:58.495507+0000","last_clean":"2023-09-18T20:43:58.495507+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:43:58.495507+0000","last_undegraded":"2023-09-18T20:43:58.495507+0000","last_fullsized":"2023-09-18T20:43:58.495507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:57:30.344645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'4969","reported_seq":9933,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.154739+0000","last_change":"2023-09-18T20:40:40.832115+0000","last_active":"2023-09-18T20:44:33.154739+0000","last_peered":"2023-09-18T20:44:33.154739+0000","last_clean":"2023-09-18T20:44:33.154739+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:44:33.154739+0000","last_undegraded":"2023-09-18T20:44:33.154739+0000","last_fullsized":"2023-09-18T20:44:33.154739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:29:42.639677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643058,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4927,"num_read_kb":19684,"num_write":4932,"num_write_kb":138346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4581","reported_seq":9040,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.453882+0000","last_change":"2023-09-18T20:40:40.790674+0000","last_active":"2023-09-18T20:43:57.453882+0000","last_peered":"2023-09-18T20:43:57.453882+0000","last_clean":"2023-09-18T20:43:57.453882+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:43:57.453882+0000","last_undegraded":"2023-09-18T20:43:57.453882+0000","last_fullsized":"2023-09-18T20:43:57.453882+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:32:31.952190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"31'4431","reported_seq":8836,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.465603+0000","last_change":"2023-09-18T20:40:40.790794+0000","last_active":"2023-09-18T20:43:57.465603+0000","last_peered":"2023-09-18T20:43:57.465603+0000","last_clean":"2023-09-18T20:43:57.465603+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:43:57.465603+0000","last_undegraded":"2023-09-18T20:43:57.465603+0000","last_fullsized":"2023-09-18T20:43:57.465603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:27:46.524729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'5099","reported_seq":10331,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.457884+0000","last_change":"2023-09-18T20:40:40.832637+0000","last_active":"2023-09-18T20:43:57.457884+0000","last_peered":"2023-09-18T20:43:57.457884+0000","last_clean":"2023-09-18T20:43:57.457884+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:43:57.457884+0000","last_undegraded":"2023-09-18T20:43:57.457884+0000","last_fullsized":"2023-09-18T20:43:57.457884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:01.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"31'6089","reported_seq":11936,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.510475+0000","last_change":"2023-09-18T20:40:40.786619+0000","last_active":"2023-09-18T20:43:57.510475+0000","last_peered":"2023-09-18T20:43:57.510475+0000","last_clean":"2023-09-18T20:43:57.510475+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:43:57.510475+0000","last_undegraded":"2023-09-18T20:43:57.510475+0000","last_fullsized":"2023-09-18T20:43:57.510475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:30:31.491319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'6927","reported_seq":13648,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.198767+0000","last_change":"2023-09-18T20:40:40.833412+0000","last_active":"2023-09-18T20:44:33.198767+0000","last_peered":"2023-09-18T20:44:33.198767+0000","last_clean":"2023-09-18T20:44:33.198767+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:44:33.198767+0000","last_undegraded":"2023-09-18T20:44:33.198767+0000","last_fullsized":"2023-09-18T20:44:33.198767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:40:52.246018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6825,"num_read_kb":26405,"num_write":6656,"num_write_kb":186153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6672","reported_seq":13399,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.450693+0000","last_change":"2023-09-18T20:40:40.785488+0000","last_active":"2023-09-18T20:43:57.450693+0000","last_peered":"2023-09-18T20:43:57.450693+0000","last_clean":"2023-09-18T20:43:57.450693+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:43:57.450693+0000","last_undegraded":"2023-09-18T20:43:57.450693+0000","last_fullsized":"2023-09-18T20:43:57.450693+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:43:31.948097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390656,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6622,"num_write_kb":190144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'5371","reported_seq":10457,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.504285+0000","last_change":"2023-09-18T20:40:40.789720+0000","last_active":"2023-09-18T20:43:58.504285+0000","last_peered":"2023-09-18T20:43:58.504285+0000","last_clean":"2023-09-18T20:43:58.504285+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:43:58.504285+0000","last_undegraded":"2023-09-18T20:43:58.504285+0000","last_fullsized":"2023-09-18T20:43:58.504285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:56:59.436258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"31'4666","reported_seq":9359,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.183709+0000","last_change":"2023-09-18T20:40:40.790245+0000","last_active":"2023-09-18T20:44:33.183709+0000","last_peered":"2023-09-18T20:44:33.183709+0000","last_clean":"2023-09-18T20:44:33.183709+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:44:33.183709+0000","last_undegraded":"2023-09-18T20:44:33.183709+0000","last_fullsized":"2023-09-18T20:44:33.183709+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:34:55.473325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4656,"num_read_kb":18896,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216316252,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":168915,"num_read_kb":672650,"num_write":169049,"num_write_kb":4770819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392638464,"data_stored":8400274104,"data_compressed":321736,"data_compressed_allocated":2179072,"data_compressed_original":9904128,"omap_allocated":0,"internal_metadata":0},"log_size":171545,"ondisk_log_size":171545,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738436,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2816096,"kb_used_data":2397784,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91555744,"statfs":{"total":96636764160,"available":93753081856,"internally_reserved":0,"allocated":2455330816,"data_stored":2456646266,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0660000000000001,"15min":0.55100000000000005},"min":{"1min":0.16700000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":3.4380000000000002,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.04,"15min":0.54900000000000004},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.5730000000000004,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.3260000000000001,"15min":0.64400000000000002},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.629,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.3160000000000001,"15min":0.65200000000000002},"min":{"1min":0.27100000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.6289999999999996,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2974192,"kb_used_data":2532264,"kb_used_omap":1,"kb_used_meta":441918,"kb_avail":91397648,"statfs":{"total":96636764160,"available":93591191552,"internally_reserved":0,"allocated":2593038336,"data_stored":2595767546,"data_compressed":58656,"data_compressed_allocated":425984,"data_compressed_original":3350528,"omap_allocated":1594,"internal_metadata":452524486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.46300000000000002,"15min":0.34699999999999998},"min":{"1min":0.18099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.59399999999999997,"15min":0.40500000000000003},"min":{"1min":0.189,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.0049999999999999,"5min":3.585,"15min":3.585},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.39200000000000002},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5489999999999999,"5min":12.382,"15min":12.382},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.53400000000000003,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":0.45700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738436,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3829464,"kb_used_data":3266768,"kb_used_omap":1,"kb_used_meta":562686,"kb_avail":90542376,"statfs":{"total":96636764160,"available":92715393024,"internally_reserved":0,"allocated":3345170432,"data_stored":3349337389,"data_compressed":164891,"data_compressed_allocated":1114112,"data_compressed_original":5488640,"omap_allocated":1594,"internal_metadata":576190918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80900000000000005,"15min":0.45700000000000002},"min":{"1min":0.23799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":5.6150000000000002,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.9079999999999999,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.94199999999999995,"15min":0.503},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.4710000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.1379999999999999,"15min":0.58399999999999996},"min":{"1min":0.26300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":5.5049999999999999,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592718848,"data_stored":2595078391,"data_compressed":52894,"data_compressed_allocated":389120,"data_compressed_original":2777088,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344850944,"data_stored":3348648234,"data_compressed":159129,"data_compressed_allocated":1077248,"data_compressed_original":4915200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455068672,"data_stored":2456547479,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:44:41.546 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:44:41.866 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:44:41.868 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:44:41.880 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2023-09-18T20:44:41.724412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216906620,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":169021,"num_read_kb":672741,"num_write":169186,"num_write_kb":4772713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171642,"ondisk_log_size":171642,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9619752,"kb_used_data":8196816,"kb_used_omap":4,"kb_used_meta":1422907,"kb_avail":273495768,"statfs":{"total":289910292480,"available":280059666432,"internally_reserved":0,"allocated":8393539584,"data_stored":8401751201,"data_compressed":333260,"data_compressed_allocated":2252800,"data_compressed_original":11051008,"omap_allocated":4782,"internal_metadata":1457057106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":208,"num_read_kb":180,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002500"},"pg_stats":[{"pgid":"3.1f","version":"31'5467","reported_seq":10804,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.490803+0000","last_change":"2023-09-18T20:40:40.833224+0000","last_active":"2023-09-18T20:43:57.490803+0000","last_peered":"2023-09-18T20:43:57.490803+0000","last_clean":"2023-09-18T20:43:57.490803+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:43:57.490803+0000","last_undegraded":"2023-09-18T20:43:57.490803+0000","last_fullsized":"2023-09-18T20:43:57.490803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:34:11.826626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4969","reported_seq":9780,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.504293+0000","last_change":"2023-09-18T20:40:40.786113+0000","last_active":"2023-09-18T20:43:57.504293+0000","last_peered":"2023-09-18T20:43:57.504293+0000","last_clean":"2023-09-18T20:43:57.504293+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:43:57.504293+0000","last_undegraded":"2023-09-18T20:43:57.504293+0000","last_fullsized":"2023-09-18T20:43:57.504293+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T20:55:47.617213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'5712","reported_seq":11417,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.449589+0000","last_change":"2023-09-18T20:40:40.785804+0000","last_active":"2023-09-18T20:43:57.449589+0000","last_peered":"2023-09-18T20:43:57.449589+0000","last_clean":"2023-09-18T20:43:57.449589+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:43:57.449589+0000","last_undegraded":"2023-09-18T20:43:57.449589+0000","last_fullsized":"2023-09-18T20:43:57.449589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:50:15.757762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614528,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5659,"num_write_kb":161724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5892","reported_seq":11756,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.200432+0000","last_change":"2023-09-18T20:40:40.790340+0000","last_active":"2023-09-18T20:44:33.200432+0000","last_peered":"2023-09-18T20:44:33.200432+0000","last_clean":"2023-09-18T20:44:33.200432+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:44:33.200432+0000","last_undegraded":"2023-09-18T20:44:33.200432+0000","last_fullsized":"2023-09-18T20:44:33.200432+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:25:56.139626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5838,"num_read_kb":23171,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5576","reported_seq":11077,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.480878+0000","last_change":"2023-09-18T20:40:40.832137+0000","last_active":"2023-09-18T20:43:57.480878+0000","last_peered":"2023-09-18T20:43:57.480878+0000","last_clean":"2023-09-18T20:43:57.480878+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:43:57.480878+0000","last_undegraded":"2023-09-18T20:43:57.480878+0000","last_fullsized":"2023-09-18T20:43:57.480878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:32:54.275405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"31'4802","reported_seq":9860,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.199077+0000","last_change":"2023-09-18T20:40:40.833200+0000","last_active":"2023-09-18T20:44:33.199077+0000","last_peered":"2023-09-18T20:44:33.199077+0000","last_clean":"2023-09-18T20:44:33.199077+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:44:33.199077+0000","last_undegraded":"2023-09-18T20:44:33.199077+0000","last_fullsized":"2023-09-18T20:44:33.199077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:06:48.959386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124866560,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5078,"num_read_kb":20357,"num_write":4767,"num_write_kb":138221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.888009+0000","last_change":"2023-09-18T20:39:18.754622+0000","last_active":"2023-09-18T20:43:27.888009+0000","last_peered":"2023-09-18T20:43:27.888009+0000","last_clean":"2023-09-18T20:43:27.888009+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:43:27.888009+0000","last_undegraded":"2023-09-18T20:43:27.888009+0000","last_fullsized":"2023-09-18T20:43:27.888009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:11:14.585598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887711+0000","last_change":"2023-09-18T20:39:18.755105+0000","last_active":"2023-09-18T20:43:27.887711+0000","last_peered":"2023-09-18T20:43:27.887711+0000","last_clean":"2023-09-18T20:43:27.887711+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:43:27.887711+0000","last_undegraded":"2023-09-18T20:43:27.887711+0000","last_fullsized":"2023-09-18T20:43:27.887711+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:09:50.066257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'4660","reported_seq":9362,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.431297+0000","last_change":"2023-09-18T20:40:40.786389+0000","last_active":"2023-09-18T20:43:57.431297+0000","last_peered":"2023-09-18T20:43:57.431297+0000","last_clean":"2023-09-18T20:43:57.431297+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:43:57.431297+0000","last_undegraded":"2023-09-18T20:43:57.431297+0000","last_fullsized":"2023-09-18T20:43:57.431297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:01:55.705787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997568,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4614,"num_write_kb":133028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'7540","reported_seq":15114,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.499606+0000","last_change":"2023-09-18T20:40:40.791026+0000","last_active":"2023-09-18T20:43:58.499606+0000","last_peered":"2023-09-18T20:43:58.499606+0000","last_clean":"2023-09-18T20:43:58.499606+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:43:58.499606+0000","last_undegraded":"2023-09-18T20:43:58.499606+0000","last_fullsized":"2023-09-18T20:43:58.499606+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:47:41.555603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.891231+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:43:27.891231+0000","last_peered":"2023-09-18T20:43:27.891231+0000","last_clean":"2023-09-18T20:43:27.891231+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:43:27.891231+0000","last_undegraded":"2023-09-18T20:43:27.891231+0000","last_fullsized":"2023-09-18T20:43:27.891231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:15:46.444320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6505","reported_seq":12883,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.464898+0000","last_change":"2023-09-18T20:40:40.831880+0000","last_active":"2023-09-18T20:43:57.464898+0000","last_peered":"2023-09-18T20:43:57.464898+0000","last_clean":"2023-09-18T20:43:57.464898+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:43:57.464898+0000","last_undegraded":"2023-09-18T20:43:57.464898+0000","last_fullsized":"2023-09-18T20:43:57.464898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:39:27.193036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.873392+0000","last_change":"2023-09-18T20:39:18.753469+0000","last_active":"2023-09-18T20:43:26.873392+0000","last_peered":"2023-09-18T20:43:26.873392+0000","last_clean":"2023-09-18T20:43:26.873392+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:43:26.873392+0000","last_undegraded":"2023-09-18T20:43:26.873392+0000","last_fullsized":"2023-09-18T20:43:26.873392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:22:56.897789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'4259","reported_seq":8454,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.452680+0000","last_change":"2023-09-18T20:40:40.790444+0000","last_active":"2023-09-18T20:43:57.452680+0000","last_peered":"2023-09-18T20:43:57.452680+0000","last_clean":"2023-09-18T20:43:57.452680+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:43:57.452680+0000","last_undegraded":"2023-09-18T20:43:57.452680+0000","last_fullsized":"2023-09-18T20:43:57.452680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:15:42.288917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":159,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.880206+0000","last_change":"2023-09-18T20:39:12.868453+0000","last_active":"2023-09-18T20:43:26.880206+0000","last_peered":"2023-09-18T20:43:26.880206+0000","last_clean":"2023-09-18T20:43:26.880206+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:43:26.880206+0000","last_undegraded":"2023-09-18T20:43:26.880206+0000","last_fullsized":"2023-09-18T20:43:26.880206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_clean_scrub_stamp":"2023-09-18T20:39:11.726727+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:32:10.120164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887509+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:43:27.887509+0000","last_peered":"2023-09-18T20:43:27.887509+0000","last_clean":"2023-09-18T20:43:27.887509+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:43:27.887509+0000","last_undegraded":"2023-09-18T20:43:27.887509+0000","last_fullsized":"2023-09-18T20:43:27.887509+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:58:59.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'5837","reported_seq":11562,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.492284+0000","last_change":"2023-09-18T20:40:40.790900+0000","last_active":"2023-09-18T20:43:58.492284+0000","last_peered":"2023-09-18T20:43:58.492284+0000","last_clean":"2023-09-18T20:43:58.492284+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:43:58.492284+0000","last_undegraded":"2023-09-18T20:43:58.492284+0000","last_fullsized":"2023-09-18T20:43:58.492284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:17:11.468556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.875264+0000","last_change":"2023-09-18T20:39:18.753462+0000","last_active":"2023-09-18T20:43:26.875264+0000","last_peered":"2023-09-18T20:43:26.875264+0000","last_clean":"2023-09-18T20:43:26.875264+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:43:26.875264+0000","last_undegraded":"2023-09-18T20:43:26.875264+0000","last_fullsized":"2023-09-18T20:43:26.875264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:13:27.835062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"31'5805","reported_seq":11502,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.500020+0000","last_change":"2023-09-18T20:40:40.832355+0000","last_active":"2023-09-18T20:43:57.500020+0000","last_peered":"2023-09-18T20:43:57.500020+0000","last_clean":"2023-09-18T20:43:57.500020+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:43:57.500020+0000","last_undegraded":"2023-09-18T20:43:57.500020+0000","last_fullsized":"2023-09-18T20:43:57.500020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:44:19.351290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.147408+0000","last_change":"2023-09-18T20:39:18.753362+0000","last_active":"2023-09-18T20:43:27.147408+0000","last_peered":"2023-09-18T20:43:27.147408+0000","last_clean":"2023-09-18T20:43:27.147408+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:43:27.147408+0000","last_undegraded":"2023-09-18T20:43:27.147408+0000","last_fullsized":"2023-09-18T20:43:27.147408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:57:37.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6170","reported_seq":12274,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.430133+0000","last_change":"2023-09-18T20:40:40.786258+0000","last_active":"2023-09-18T20:43:57.430133+0000","last_peered":"2023-09-18T20:43:57.430133+0000","last_clean":"2023-09-18T20:43:57.430133+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:43:57.430133+0000","last_undegraded":"2023-09-18T20:43:57.430133+0000","last_fullsized":"2023-09-18T20:43:57.430133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:33:56.555234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.889508+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:43:27.889508+0000","last_peered":"2023-09-18T20:43:27.889508+0000","last_clean":"2023-09-18T20:43:27.889508+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:43:27.889508+0000","last_undegraded":"2023-09-18T20:43:27.889508+0000","last_fullsized":"2023-09-18T20:43:27.889508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:46:11.076182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"31'5238","reported_seq":10420,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486825+0000","last_change":"2023-09-18T20:40:40.785694+0000","last_active":"2023-09-18T20:43:57.486825+0000","last_peered":"2023-09-18T20:43:57.486825+0000","last_clean":"2023-09-18T20:43:57.486825+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:43:57.486825+0000","last_undegraded":"2023-09-18T20:43:57.486825+0000","last_fullsized":"2023-09-18T20:43:57.486825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:48:25.383550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"31'3820","reported_seq":7550,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.197751+0000","last_change":"2023-09-18T20:40:40.831884+0000","last_active":"2023-09-18T20:44:33.197751+0000","last_peered":"2023-09-18T20:44:33.197751+0000","last_clean":"2023-09-18T20:44:33.197751+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:44:33.197751+0000","last_undegraded":"2023-09-18T20:44:33.197751+0000","last_fullsized":"2023-09-18T20:44:33.197751+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:26:57.735585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3828,"num_read_kb":14459,"num_write":3566,"num_write_kb":100156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"31'7146","reported_seq":13922,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.195586+0000","last_change":"2023-09-18T20:40:40.832373+0000","last_active":"2023-09-18T20:44:33.195586+0000","last_peered":"2023-09-18T20:44:33.195586+0000","last_clean":"2023-09-18T20:44:33.195586+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:44:33.195586+0000","last_undegraded":"2023-09-18T20:44:33.195586+0000","last_fullsized":"2023-09-18T20:44:33.195586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:23:31.794226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":6873,"num_read_kb":27082,"num_write":6864,"num_write_kb":191656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'3581","reported_seq":7123,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.402162+0000","last_change":"2023-09-18T20:40:40.790576+0000","last_active":"2023-09-18T20:43:57.402162+0000","last_peered":"2023-09-18T20:43:57.402162+0000","last_clean":"2023-09-18T20:43:57.402162+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:43:57.402162+0000","last_undegraded":"2023-09-18T20:43:57.402162+0000","last_fullsized":"2023-09-18T20:43:57.402162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:13:44.471080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'4630","reported_seq":9019,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.199745+0000","last_change":"2023-09-18T20:40:40.832593+0000","last_active":"2023-09-18T20:44:33.199745+0000","last_peered":"2023-09-18T20:44:33.199745+0000","last_clean":"2023-09-18T20:44:33.199745+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:44:33.199745+0000","last_undegraded":"2023-09-18T20:44:33.199745+0000","last_fullsized":"2023-09-18T20:44:33.199745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:21:52.199914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108081152,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4486,"num_read_kb":17534,"num_write":4368,"num_write_kb":120480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"31'4114","reported_seq":8338,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.454474+0000","last_change":"2023-09-18T20:40:40.791131+0000","last_active":"2023-09-18T20:43:57.454474+0000","last_peered":"2023-09-18T20:43:57.454474+0000","last_clean":"2023-09-18T20:43:57.454474+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:43:57.454474+0000","last_undegraded":"2023-09-18T20:43:57.454474+0000","last_fullsized":"2023-09-18T20:43:57.454474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:05:41.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"31'6084","reported_seq":12190,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.488321+0000","last_change":"2023-09-18T20:40:40.786504+0000","last_active":"2023-09-18T20:43:57.488321+0000","last_peered":"2023-09-18T20:43:57.488321+0000","last_clean":"2023-09-18T20:43:57.488321+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:43:57.488321+0000","last_undegraded":"2023-09-18T20:43:57.488321+0000","last_fullsized":"2023-09-18T20:43:57.488321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:22:28.068618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'4159","reported_seq":8163,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.196731+0000","last_change":"2023-09-18T20:40:40.833389+0000","last_active":"2023-09-18T20:44:33.196731+0000","last_peered":"2023-09-18T20:44:33.196731+0000","last_clean":"2023-09-18T20:44:33.196731+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:44:33.196731+0000","last_undegraded":"2023-09-18T20:44:33.196731+0000","last_fullsized":"2023-09-18T20:44:33.196731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:27.656176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4108,"num_read_kb":15537,"num_write":3893,"num_write_kb":109673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"31'4527","reported_seq":9042,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486241+0000","last_change":"2023-09-18T20:40:40.789756+0000","last_active":"2023-09-18T20:43:57.486241+0000","last_peered":"2023-09-18T20:43:57.486241+0000","last_clean":"2023-09-18T20:43:57.486241+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:43:57.486241+0000","last_undegraded":"2023-09-18T20:43:57.486241+0000","last_fullsized":"2023-09-18T20:43:57.486241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:33:32.442279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'6247","reported_seq":12323,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.495507+0000","last_change":"2023-09-18T20:40:40.785973+0000","last_active":"2023-09-18T20:43:58.495507+0000","last_peered":"2023-09-18T20:43:58.495507+0000","last_clean":"2023-09-18T20:43:58.495507+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:43:58.495507+0000","last_undegraded":"2023-09-18T20:43:58.495507+0000","last_fullsized":"2023-09-18T20:43:58.495507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:57:30.344645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'4969","reported_seq":9933,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.154739+0000","last_change":"2023-09-18T20:40:40.832115+0000","last_active":"2023-09-18T20:44:33.154739+0000","last_peered":"2023-09-18T20:44:33.154739+0000","last_clean":"2023-09-18T20:44:33.154739+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:44:33.154739+0000","last_undegraded":"2023-09-18T20:44:33.154739+0000","last_fullsized":"2023-09-18T20:44:33.154739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:29:42.639677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643058,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4927,"num_read_kb":19684,"num_write":4932,"num_write_kb":138346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4581","reported_seq":9040,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.453882+0000","last_change":"2023-09-18T20:40:40.790674+0000","last_active":"2023-09-18T20:43:57.453882+0000","last_peered":"2023-09-18T20:43:57.453882+0000","last_clean":"2023-09-18T20:43:57.453882+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:43:57.453882+0000","last_undegraded":"2023-09-18T20:43:57.453882+0000","last_fullsized":"2023-09-18T20:43:57.453882+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:32:31.952190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"31'4431","reported_seq":8836,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.465603+0000","last_change":"2023-09-18T20:40:40.790794+0000","last_active":"2023-09-18T20:43:57.465603+0000","last_peered":"2023-09-18T20:43:57.465603+0000","last_clean":"2023-09-18T20:43:57.465603+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:43:57.465603+0000","last_undegraded":"2023-09-18T20:43:57.465603+0000","last_fullsized":"2023-09-18T20:43:57.465603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:27:46.524729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'5099","reported_seq":10331,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.457884+0000","last_change":"2023-09-18T20:40:40.832637+0000","last_active":"2023-09-18T20:43:57.457884+0000","last_peered":"2023-09-18T20:43:57.457884+0000","last_clean":"2023-09-18T20:43:57.457884+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:43:57.457884+0000","last_undegraded":"2023-09-18T20:43:57.457884+0000","last_fullsized":"2023-09-18T20:43:57.457884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:01.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"31'6089","reported_seq":11936,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.510475+0000","last_change":"2023-09-18T20:40:40.786619+0000","last_active":"2023-09-18T20:43:57.510475+0000","last_peered":"2023-09-18T20:43:57.510475+0000","last_clean":"2023-09-18T20:43:57.510475+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:43:57.510475+0000","last_undegraded":"2023-09-18T20:43:57.510475+0000","last_fullsized":"2023-09-18T20:43:57.510475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:30:31.491319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'6927","reported_seq":13648,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.198767+0000","last_change":"2023-09-18T20:40:40.833412+0000","last_active":"2023-09-18T20:44:33.198767+0000","last_peered":"2023-09-18T20:44:33.198767+0000","last_clean":"2023-09-18T20:44:33.198767+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:44:33.198767+0000","last_undegraded":"2023-09-18T20:44:33.198767+0000","last_fullsized":"2023-09-18T20:44:33.198767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:40:52.246018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6825,"num_read_kb":26405,"num_write":6656,"num_write_kb":186153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6672","reported_seq":13399,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.450693+0000","last_change":"2023-09-18T20:40:40.785488+0000","last_active":"2023-09-18T20:43:57.450693+0000","last_peered":"2023-09-18T20:43:57.450693+0000","last_clean":"2023-09-18T20:43:57.450693+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:43:57.450693+0000","last_undegraded":"2023-09-18T20:43:57.450693+0000","last_fullsized":"2023-09-18T20:43:57.450693+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:43:31.948097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390656,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6622,"num_write_kb":190144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'5371","reported_seq":10457,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.504285+0000","last_change":"2023-09-18T20:40:40.789720+0000","last_active":"2023-09-18T20:43:58.504285+0000","last_peered":"2023-09-18T20:43:58.504285+0000","last_clean":"2023-09-18T20:43:58.504285+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:43:58.504285+0000","last_undegraded":"2023-09-18T20:43:58.504285+0000","last_fullsized":"2023-09-18T20:43:58.504285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:56:59.436258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"31'4666","reported_seq":9359,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.183709+0000","last_change":"2023-09-18T20:40:40.790245+0000","last_active":"2023-09-18T20:44:33.183709+0000","last_peered":"2023-09-18T20:44:33.183709+0000","last_clean":"2023-09-18T20:44:33.183709+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:44:33.183709+0000","last_undegraded":"2023-09-18T20:44:33.183709+0000","last_fullsized":"2023-09-18T20:44:33.183709+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:34:55.473325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4656,"num_read_kb":18896,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216316252,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":168915,"num_read_kb":672650,"num_write":169049,"num_write_kb":4770819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392638464,"data_stored":8400274104,"data_compressed":321736,"data_compressed_allocated":2179072,"data_compressed_original":9904128,"omap_allocated":0,"internal_metadata":0},"log_size":171545,"ondisk_log_size":171545,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738437,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2816096,"kb_used_data":2397784,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91555744,"statfs":{"total":96636764160,"available":93753081856,"internally_reserved":0,"allocated":2455330816,"data_stored":2456646266,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0660000000000001,"15min":0.55100000000000005},"min":{"1min":0.16700000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":3.4380000000000002,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.04,"15min":0.54900000000000004},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.5730000000000004,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.3260000000000001,"15min":0.64400000000000002},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.629,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.3160000000000001,"15min":0.65200000000000002},"min":{"1min":0.27100000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.6289999999999996,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2974192,"kb_used_data":2532264,"kb_used_omap":1,"kb_used_meta":441918,"kb_avail":91397648,"statfs":{"total":96636764160,"available":93591191552,"internally_reserved":0,"allocated":2593038336,"data_stored":2595767546,"data_compressed":58656,"data_compressed_allocated":425984,"data_compressed_original":3350528,"omap_allocated":1594,"internal_metadata":452524486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.46300000000000002,"15min":0.34699999999999998},"min":{"1min":0.18099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.59399999999999997,"15min":0.40500000000000003},"min":{"1min":0.189,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.0049999999999999,"5min":3.585,"15min":3.585},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.39200000000000002},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5489999999999999,"5min":12.382,"15min":12.382},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.53400000000000003,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":0.45700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738436,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3829464,"kb_used_data":3266768,"kb_used_omap":1,"kb_used_meta":562686,"kb_avail":90542376,"statfs":{"total":96636764160,"available":92715393024,"internally_reserved":0,"allocated":3345170432,"data_stored":3349337389,"data_compressed":164891,"data_compressed_allocated":1114112,"data_compressed_original":5488640,"omap_allocated":1594,"internal_metadata":576190918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80900000000000005,"15min":0.45700000000000002},"min":{"1min":0.23799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":5.6150000000000002,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.9079999999999999,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.94199999999999995,"15min":0.503},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.4710000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.1379999999999999,"15min":0.58399999999999996},"min":{"1min":0.26300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":5.5049999999999999,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592718848,"data_stored":2595078391,"data_compressed":52894,"data_compressed_allocated":389120,"data_compressed_original":2777088,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344850944,"data_stored":3348648234,"data_compressed":159129,"data_compressed_allocated":1077248,"data_compressed_original":4915200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455068672,"data_stored":2456547479,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:44:41.883 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-18T20:44:41.883 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:44:42.205 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:44:42.207 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:44:42.218 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2023-09-18T20:44:41.724412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216906620,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":169021,"num_read_kb":672741,"num_write":169186,"num_write_kb":4772713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171642,"ondisk_log_size":171642,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9619752,"kb_used_data":8196816,"kb_used_omap":4,"kb_used_meta":1422907,"kb_avail":273495768,"statfs":{"total":289910292480,"available":280059666432,"internally_reserved":0,"allocated":8393539584,"data_stored":8401751201,"data_compressed":333260,"data_compressed_allocated":2252800,"data_compressed_original":11051008,"omap_allocated":4782,"internal_metadata":1457057106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":208,"num_read_kb":180,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002500"},"pg_stats":[{"pgid":"3.1f","version":"31'5467","reported_seq":10804,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.490803+0000","last_change":"2023-09-18T20:40:40.833224+0000","last_active":"2023-09-18T20:43:57.490803+0000","last_peered":"2023-09-18T20:43:57.490803+0000","last_clean":"2023-09-18T20:43:57.490803+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:43:57.490803+0000","last_undegraded":"2023-09-18T20:43:57.490803+0000","last_fullsized":"2023-09-18T20:43:57.490803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:34:11.826626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4969","reported_seq":9780,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.504293+0000","last_change":"2023-09-18T20:40:40.786113+0000","last_active":"2023-09-18T20:43:57.504293+0000","last_peered":"2023-09-18T20:43:57.504293+0000","last_clean":"2023-09-18T20:43:57.504293+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:43:57.504293+0000","last_undegraded":"2023-09-18T20:43:57.504293+0000","last_fullsized":"2023-09-18T20:43:57.504293+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T20:55:47.617213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'5712","reported_seq":11417,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.449589+0000","last_change":"2023-09-18T20:40:40.785804+0000","last_active":"2023-09-18T20:43:57.449589+0000","last_peered":"2023-09-18T20:43:57.449589+0000","last_clean":"2023-09-18T20:43:57.449589+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:43:57.449589+0000","last_undegraded":"2023-09-18T20:43:57.449589+0000","last_fullsized":"2023-09-18T20:43:57.449589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:50:15.757762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614528,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5659,"num_write_kb":161724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5892","reported_seq":11756,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.200432+0000","last_change":"2023-09-18T20:40:40.790340+0000","last_active":"2023-09-18T20:44:33.200432+0000","last_peered":"2023-09-18T20:44:33.200432+0000","last_clean":"2023-09-18T20:44:33.200432+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:44:33.200432+0000","last_undegraded":"2023-09-18T20:44:33.200432+0000","last_fullsized":"2023-09-18T20:44:33.200432+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:25:56.139626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5838,"num_read_kb":23171,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5576","reported_seq":11077,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.480878+0000","last_change":"2023-09-18T20:40:40.832137+0000","last_active":"2023-09-18T20:43:57.480878+0000","last_peered":"2023-09-18T20:43:57.480878+0000","last_clean":"2023-09-18T20:43:57.480878+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:43:57.480878+0000","last_undegraded":"2023-09-18T20:43:57.480878+0000","last_fullsized":"2023-09-18T20:43:57.480878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:32:54.275405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"31'4802","reported_seq":9860,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.199077+0000","last_change":"2023-09-18T20:40:40.833200+0000","last_active":"2023-09-18T20:44:33.199077+0000","last_peered":"2023-09-18T20:44:33.199077+0000","last_clean":"2023-09-18T20:44:33.199077+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:44:33.199077+0000","last_undegraded":"2023-09-18T20:44:33.199077+0000","last_fullsized":"2023-09-18T20:44:33.199077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:06:48.959386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124866560,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5078,"num_read_kb":20357,"num_write":4767,"num_write_kb":138221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.888009+0000","last_change":"2023-09-18T20:39:18.754622+0000","last_active":"2023-09-18T20:43:27.888009+0000","last_peered":"2023-09-18T20:43:27.888009+0000","last_clean":"2023-09-18T20:43:27.888009+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:43:27.888009+0000","last_undegraded":"2023-09-18T20:43:27.888009+0000","last_fullsized":"2023-09-18T20:43:27.888009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:11:14.585598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887711+0000","last_change":"2023-09-18T20:39:18.755105+0000","last_active":"2023-09-18T20:43:27.887711+0000","last_peered":"2023-09-18T20:43:27.887711+0000","last_clean":"2023-09-18T20:43:27.887711+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:43:27.887711+0000","last_undegraded":"2023-09-18T20:43:27.887711+0000","last_fullsized":"2023-09-18T20:43:27.887711+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:09:50.066257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'4660","reported_seq":9362,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.431297+0000","last_change":"2023-09-18T20:40:40.786389+0000","last_active":"2023-09-18T20:43:57.431297+0000","last_peered":"2023-09-18T20:43:57.431297+0000","last_clean":"2023-09-18T20:43:57.431297+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:43:57.431297+0000","last_undegraded":"2023-09-18T20:43:57.431297+0000","last_fullsized":"2023-09-18T20:43:57.431297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:01:55.705787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997568,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4614,"num_write_kb":133028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'7540","reported_seq":15114,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.499606+0000","last_change":"2023-09-18T20:40:40.791026+0000","last_active":"2023-09-18T20:43:58.499606+0000","last_peered":"2023-09-18T20:43:58.499606+0000","last_clean":"2023-09-18T20:43:58.499606+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:43:58.499606+0000","last_undegraded":"2023-09-18T20:43:58.499606+0000","last_fullsized":"2023-09-18T20:43:58.499606+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:47:41.555603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.891231+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:43:27.891231+0000","last_peered":"2023-09-18T20:43:27.891231+0000","last_clean":"2023-09-18T20:43:27.891231+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:43:27.891231+0000","last_undegraded":"2023-09-18T20:43:27.891231+0000","last_fullsized":"2023-09-18T20:43:27.891231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:15:46.444320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6505","reported_seq":12883,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.464898+0000","last_change":"2023-09-18T20:40:40.831880+0000","last_active":"2023-09-18T20:43:57.464898+0000","last_peered":"2023-09-18T20:43:57.464898+0000","last_clean":"2023-09-18T20:43:57.464898+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:43:57.464898+0000","last_undegraded":"2023-09-18T20:43:57.464898+0000","last_fullsized":"2023-09-18T20:43:57.464898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:39:27.193036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.873392+0000","last_change":"2023-09-18T20:39:18.753469+0000","last_active":"2023-09-18T20:43:26.873392+0000","last_peered":"2023-09-18T20:43:26.873392+0000","last_clean":"2023-09-18T20:43:26.873392+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:43:26.873392+0000","last_undegraded":"2023-09-18T20:43:26.873392+0000","last_fullsized":"2023-09-18T20:43:26.873392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:22:56.897789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'4259","reported_seq":8454,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.452680+0000","last_change":"2023-09-18T20:40:40.790444+0000","last_active":"2023-09-18T20:43:57.452680+0000","last_peered":"2023-09-18T20:43:57.452680+0000","last_clean":"2023-09-18T20:43:57.452680+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:43:57.452680+0000","last_undegraded":"2023-09-18T20:43:57.452680+0000","last_fullsized":"2023-09-18T20:43:57.452680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:15:42.288917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":159,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.880206+0000","last_change":"2023-09-18T20:39:12.868453+0000","last_active":"2023-09-18T20:43:26.880206+0000","last_peered":"2023-09-18T20:43:26.880206+0000","last_clean":"2023-09-18T20:43:26.880206+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:43:26.880206+0000","last_undegraded":"2023-09-18T20:43:26.880206+0000","last_fullsized":"2023-09-18T20:43:26.880206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_clean_scrub_stamp":"2023-09-18T20:39:11.726727+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:32:10.120164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887509+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:43:27.887509+0000","last_peered":"2023-09-18T20:43:27.887509+0000","last_clean":"2023-09-18T20:43:27.887509+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:43:27.887509+0000","last_undegraded":"2023-09-18T20:43:27.887509+0000","last_fullsized":"2023-09-18T20:43:27.887509+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:58:59.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'5837","reported_seq":11562,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.492284+0000","last_change":"2023-09-18T20:40:40.790900+0000","last_active":"2023-09-18T20:43:58.492284+0000","last_peered":"2023-09-18T20:43:58.492284+0000","last_clean":"2023-09-18T20:43:58.492284+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:43:58.492284+0000","last_undegraded":"2023-09-18T20:43:58.492284+0000","last_fullsized":"2023-09-18T20:43:58.492284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:17:11.468556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.875264+0000","last_change":"2023-09-18T20:39:18.753462+0000","last_active":"2023-09-18T20:43:26.875264+0000","last_peered":"2023-09-18T20:43:26.875264+0000","last_clean":"2023-09-18T20:43:26.875264+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:43:26.875264+0000","last_undegraded":"2023-09-18T20:43:26.875264+0000","last_fullsized":"2023-09-18T20:43:26.875264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:13:27.835062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"31'5805","reported_seq":11502,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.500020+0000","last_change":"2023-09-18T20:40:40.832355+0000","last_active":"2023-09-18T20:43:57.500020+0000","last_peered":"2023-09-18T20:43:57.500020+0000","last_clean":"2023-09-18T20:43:57.500020+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:43:57.500020+0000","last_undegraded":"2023-09-18T20:43:57.500020+0000","last_fullsized":"2023-09-18T20:43:57.500020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:44:19.351290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.147408+0000","last_change":"2023-09-18T20:39:18.753362+0000","last_active":"2023-09-18T20:43:27.147408+0000","last_peered":"2023-09-18T20:43:27.147408+0000","last_clean":"2023-09-18T20:43:27.147408+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:43:27.147408+0000","last_undegraded":"2023-09-18T20:43:27.147408+0000","last_fullsized":"2023-09-18T20:43:27.147408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:57:37.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6170","reported_seq":12274,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.430133+0000","last_change":"2023-09-18T20:40:40.786258+0000","last_active":"2023-09-18T20:43:57.430133+0000","last_peered":"2023-09-18T20:43:57.430133+0000","last_clean":"2023-09-18T20:43:57.430133+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:43:57.430133+0000","last_undegraded":"2023-09-18T20:43:57.430133+0000","last_fullsized":"2023-09-18T20:43:57.430133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:33:56.555234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.889508+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:43:27.889508+0000","last_peered":"2023-09-18T20:43:27.889508+0000","last_clean":"2023-09-18T20:43:27.889508+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:43:27.889508+0000","last_undegraded":"2023-09-18T20:43:27.889508+0000","last_fullsized":"2023-09-18T20:43:27.889508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:46:11.076182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"31'5238","reported_seq":10420,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486825+0000","last_change":"2023-09-18T20:40:40.785694+0000","last_active":"2023-09-18T20:43:57.486825+0000","last_peered":"2023-09-18T20:43:57.486825+0000","last_clean":"2023-09-18T20:43:57.486825+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:43:57.486825+0000","last_undegraded":"2023-09-18T20:43:57.486825+0000","last_fullsized":"2023-09-18T20:43:57.486825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:48:25.383550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"31'3820","reported_seq":7550,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.197751+0000","last_change":"2023-09-18T20:40:40.831884+0000","last_active":"2023-09-18T20:44:33.197751+0000","last_peered":"2023-09-18T20:44:33.197751+0000","last_clean":"2023-09-18T20:44:33.197751+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:44:33.197751+0000","last_undegraded":"2023-09-18T20:44:33.197751+0000","last_fullsized":"2023-09-18T20:44:33.197751+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:26:57.735585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3828,"num_read_kb":14459,"num_write":3566,"num_write_kb":100156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"31'7146","reported_seq":13922,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.195586+0000","last_change":"2023-09-18T20:40:40.832373+0000","last_active":"2023-09-18T20:44:33.195586+0000","last_peered":"2023-09-18T20:44:33.195586+0000","last_clean":"2023-09-18T20:44:33.195586+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:44:33.195586+0000","last_undegraded":"2023-09-18T20:44:33.195586+0000","last_fullsized":"2023-09-18T20:44:33.195586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:23:31.794226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":6873,"num_read_kb":27082,"num_write":6864,"num_write_kb":191656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'3581","reported_seq":7123,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.402162+0000","last_change":"2023-09-18T20:40:40.790576+0000","last_active":"2023-09-18T20:43:57.402162+0000","last_peered":"2023-09-18T20:43:57.402162+0000","last_clean":"2023-09-18T20:43:57.402162+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:43:57.402162+0000","last_undegraded":"2023-09-18T20:43:57.402162+0000","last_fullsized":"2023-09-18T20:43:57.402162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:13:44.471080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'4630","reported_seq":9019,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.199745+0000","last_change":"2023-09-18T20:40:40.832593+0000","last_active":"2023-09-18T20:44:33.199745+0000","last_peered":"2023-09-18T20:44:33.199745+0000","last_clean":"2023-09-18T20:44:33.199745+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:44:33.199745+0000","last_undegraded":"2023-09-18T20:44:33.199745+0000","last_fullsized":"2023-09-18T20:44:33.199745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:21:52.199914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108081152,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4486,"num_read_kb":17534,"num_write":4368,"num_write_kb":120480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"31'4114","reported_seq":8338,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.454474+0000","last_change":"2023-09-18T20:40:40.791131+0000","last_active":"2023-09-18T20:43:57.454474+0000","last_peered":"2023-09-18T20:43:57.454474+0000","last_clean":"2023-09-18T20:43:57.454474+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:43:57.454474+0000","last_undegraded":"2023-09-18T20:43:57.454474+0000","last_fullsized":"2023-09-18T20:43:57.454474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:05:41.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"31'6084","reported_seq":12190,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.488321+0000","last_change":"2023-09-18T20:40:40.786504+0000","last_active":"2023-09-18T20:43:57.488321+0000","last_peered":"2023-09-18T20:43:57.488321+0000","last_clean":"2023-09-18T20:43:57.488321+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:43:57.488321+0000","last_undegraded":"2023-09-18T20:43:57.488321+0000","last_fullsized":"2023-09-18T20:43:57.488321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:22:28.068618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'4159","reported_seq":8163,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.196731+0000","last_change":"2023-09-18T20:40:40.833389+0000","last_active":"2023-09-18T20:44:33.196731+0000","last_peered":"2023-09-18T20:44:33.196731+0000","last_clean":"2023-09-18T20:44:33.196731+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:44:33.196731+0000","last_undegraded":"2023-09-18T20:44:33.196731+0000","last_fullsized":"2023-09-18T20:44:33.196731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:27.656176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4108,"num_read_kb":15537,"num_write":3893,"num_write_kb":109673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"31'4527","reported_seq":9042,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486241+0000","last_change":"2023-09-18T20:40:40.789756+0000","last_active":"2023-09-18T20:43:57.486241+0000","last_peered":"2023-09-18T20:43:57.486241+0000","last_clean":"2023-09-18T20:43:57.486241+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:43:57.486241+0000","last_undegraded":"2023-09-18T20:43:57.486241+0000","last_fullsized":"2023-09-18T20:43:57.486241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:33:32.442279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'6247","reported_seq":12323,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.495507+0000","last_change":"2023-09-18T20:40:40.785973+0000","last_active":"2023-09-18T20:43:58.495507+0000","last_peered":"2023-09-18T20:43:58.495507+0000","last_clean":"2023-09-18T20:43:58.495507+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:43:58.495507+0000","last_undegraded":"2023-09-18T20:43:58.495507+0000","last_fullsized":"2023-09-18T20:43:58.495507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:57:30.344645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'4969","reported_seq":9933,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.154739+0000","last_change":"2023-09-18T20:40:40.832115+0000","last_active":"2023-09-18T20:44:33.154739+0000","last_peered":"2023-09-18T20:44:33.154739+0000","last_clean":"2023-09-18T20:44:33.154739+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:44:33.154739+0000","last_undegraded":"2023-09-18T20:44:33.154739+0000","last_fullsized":"2023-09-18T20:44:33.154739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:29:42.639677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643058,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4927,"num_read_kb":19684,"num_write":4932,"num_write_kb":138346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4581","reported_seq":9040,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.453882+0000","last_change":"2023-09-18T20:40:40.790674+0000","last_active":"2023-09-18T20:43:57.453882+0000","last_peered":"2023-09-18T20:43:57.453882+0000","last_clean":"2023-09-18T20:43:57.453882+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:43:57.453882+0000","last_undegraded":"2023-09-18T20:43:57.453882+0000","last_fullsized":"2023-09-18T20:43:57.453882+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:32:31.952190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"31'4431","reported_seq":8836,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.465603+0000","last_change":"2023-09-18T20:40:40.790794+0000","last_active":"2023-09-18T20:43:57.465603+0000","last_peered":"2023-09-18T20:43:57.465603+0000","last_clean":"2023-09-18T20:43:57.465603+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:43:57.465603+0000","last_undegraded":"2023-09-18T20:43:57.465603+0000","last_fullsized":"2023-09-18T20:43:57.465603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:27:46.524729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'5099","reported_seq":10331,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.457884+0000","last_change":"2023-09-18T20:40:40.832637+0000","last_active":"2023-09-18T20:43:57.457884+0000","last_peered":"2023-09-18T20:43:57.457884+0000","last_clean":"2023-09-18T20:43:57.457884+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:43:57.457884+0000","last_undegraded":"2023-09-18T20:43:57.457884+0000","last_fullsized":"2023-09-18T20:43:57.457884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:01.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"31'6089","reported_seq":11936,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.510475+0000","last_change":"2023-09-18T20:40:40.786619+0000","last_active":"2023-09-18T20:43:57.510475+0000","last_peered":"2023-09-18T20:43:57.510475+0000","last_clean":"2023-09-18T20:43:57.510475+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:43:57.510475+0000","last_undegraded":"2023-09-18T20:43:57.510475+0000","last_fullsized":"2023-09-18T20:43:57.510475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:30:31.491319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'6927","reported_seq":13648,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.198767+0000","last_change":"2023-09-18T20:40:40.833412+0000","last_active":"2023-09-18T20:44:33.198767+0000","last_peered":"2023-09-18T20:44:33.198767+0000","last_clean":"2023-09-18T20:44:33.198767+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:44:33.198767+0000","last_undegraded":"2023-09-18T20:44:33.198767+0000","last_fullsized":"2023-09-18T20:44:33.198767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:40:52.246018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6825,"num_read_kb":26405,"num_write":6656,"num_write_kb":186153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6672","reported_seq":13399,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.450693+0000","last_change":"2023-09-18T20:40:40.785488+0000","last_active":"2023-09-18T20:43:57.450693+0000","last_peered":"2023-09-18T20:43:57.450693+0000","last_clean":"2023-09-18T20:43:57.450693+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:43:57.450693+0000","last_undegraded":"2023-09-18T20:43:57.450693+0000","last_fullsized":"2023-09-18T20:43:57.450693+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:43:31.948097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390656,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6622,"num_write_kb":190144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'5371","reported_seq":10457,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.504285+0000","last_change":"2023-09-18T20:40:40.789720+0000","last_active":"2023-09-18T20:43:58.504285+0000","last_peered":"2023-09-18T20:43:58.504285+0000","last_clean":"2023-09-18T20:43:58.504285+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:43:58.504285+0000","last_undegraded":"2023-09-18T20:43:58.504285+0000","last_fullsized":"2023-09-18T20:43:58.504285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:56:59.436258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"31'4666","reported_seq":9359,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.183709+0000","last_change":"2023-09-18T20:40:40.790245+0000","last_active":"2023-09-18T20:44:33.183709+0000","last_peered":"2023-09-18T20:44:33.183709+0000","last_clean":"2023-09-18T20:44:33.183709+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:44:33.183709+0000","last_undegraded":"2023-09-18T20:44:33.183709+0000","last_fullsized":"2023-09-18T20:44:33.183709+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:34:55.473325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4656,"num_read_kb":18896,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216316252,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":168915,"num_read_kb":672650,"num_write":169049,"num_write_kb":4770819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392638464,"data_stored":8400274104,"data_compressed":321736,"data_compressed_allocated":2179072,"data_compressed_original":9904128,"omap_allocated":0,"internal_metadata":0},"log_size":171545,"ondisk_log_size":171545,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738437,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2816096,"kb_used_data":2397784,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91555744,"statfs":{"total":96636764160,"available":93753081856,"internally_reserved":0,"allocated":2455330816,"data_stored":2456646266,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0660000000000001,"15min":0.55100000000000005},"min":{"1min":0.16700000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":3.4380000000000002,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.04,"15min":0.54900000000000004},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.5730000000000004,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.3260000000000001,"15min":0.64400000000000002},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.629,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.3160000000000001,"15min":0.65200000000000002},"min":{"1min":0.27100000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.6289999999999996,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2974192,"kb_used_data":2532264,"kb_used_omap":1,"kb_used_meta":441918,"kb_avail":91397648,"statfs":{"total":96636764160,"available":93591191552,"internally_reserved":0,"allocated":2593038336,"data_stored":2595767546,"data_compressed":58656,"data_compressed_allocated":425984,"data_compressed_original":3350528,"omap_allocated":1594,"internal_metadata":452524486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.46300000000000002,"15min":0.34699999999999998},"min":{"1min":0.18099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.59399999999999997,"15min":0.40500000000000003},"min":{"1min":0.189,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.0049999999999999,"5min":3.585,"15min":3.585},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.39200000000000002},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5489999999999999,"5min":12.382,"15min":12.382},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.53400000000000003,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":0.45700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738436,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3829464,"kb_used_data":3266768,"kb_used_omap":1,"kb_used_meta":562686,"kb_avail":90542376,"statfs":{"total":96636764160,"available":92715393024,"internally_reserved":0,"allocated":3345170432,"data_stored":3349337389,"data_compressed":164891,"data_compressed_allocated":1114112,"data_compressed_original":5488640,"omap_allocated":1594,"internal_metadata":576190918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80900000000000005,"15min":0.45700000000000002},"min":{"1min":0.23799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":5.6150000000000002,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.9079999999999999,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.94199999999999995,"15min":0.503},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.4710000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.1379999999999999,"15min":0.58399999999999996},"min":{"1min":0.26300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":5.5049999999999999,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592718848,"data_stored":2595078391,"data_compressed":52894,"data_compressed_allocated":389120,"data_compressed_original":2777088,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344850944,"data_stored":3348648234,"data_compressed":159129,"data_compressed_allocated":1077248,"data_compressed_original":4915200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455068672,"data_stored":2456547479,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:44:42.222 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-18T20:44:42.543 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:44:42.544 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":31,"fsid":"6b7573aa-3580-401f-a189-159cf58e6075","created":"2023-09-18T20:39:01.734028+0000","modified":"2023-09-18T20:43:27.875636+0000","last_up_change":"2023-09-18T20:39:10.658176+0000","last_in_change":"2023-09-18T20:39:02.736890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-18T20:39:11.677284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-18T20:39:17.094949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"rbd2","create_time":"2023-09-18T20:40:37.170980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":8,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2738e12b-7bac-4194-8b8f-86cacbd2308f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6801","nonce":1950803321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6803","nonce":1950803321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6807","nonce":1950803321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1950803321},{"type":"v1","addr":"172.21.15.149:6805","nonce":1950803321}]},"public_addr":"172.21.15.149:6801/1950803321","cluster_addr":"172.21.15.149:6803/1950803321","heartbeat_back_addr":"172.21.15.149:6807/1950803321","heartbeat_front_addr":"172.21.15.149:6805/1950803321","state":["exists","up"]},{"osd":1,"uuid":"e69b711f-10f7-4911-a025-cab999fb19b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6812","nonce":1550375764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6817","nonce":1550375764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6823","nonce":1550375764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1550375764},{"type":"v1","addr":"172.21.15.149:6821","nonce":1550375764}]},"public_addr":"172.21.15.149:6812/1550375764","cluster_addr":"172.21.15.149:6817/1550375764","heartbeat_back_addr":"172.21.15.149:6823/1550375764","heartbeat_front_addr":"172.21.15.149:6821/1550375764","state":["exists","up"]},{"osd":2,"uuid":"978dafe6-0507-4167-89fe-b27a8496f2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6809","nonce":2507427811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6813","nonce":2507427811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6819","nonce":2507427811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2507427811},{"type":"v1","addr":"172.21.15.149:6816","nonce":2507427811}]},"public_addr":"172.21.15.149:6809/2507427811","cluster_addr":"172.21.15.149:6813/2507427811","heartbeat_back_addr":"172.21.15.149:6819/2507427811","heartbeat_front_addr":"172.21.15.149:6816/2507427811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.932185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.979717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-18T20:39:07.938873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.149:0/1940331705":"2023-09-18T21:43:26.779898+0000","172.21.15.149:0/2711493074":"2023-09-18T21:42:36.817088+0000","172.21.15.149:0/2882024060":"2023-09-18T21:40:50.470956+0000","172.21.15.149:0/4070770118":"2023-09-18T21:41:25.852909+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-18T20:44:43.558 INFO:tasks.ceph:Scrubbing osd.0 2023-09-18T20:44:43.559 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-18T20:44:43.719 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2023-09-18T20:44:43.719 INFO:teuthology.orchestra.run.smithi149.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-18T20:44:43.719 INFO:teuthology.orchestra.run.smithi149.stdout:} 2023-09-18T20:44:43.731 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-18T20:44:44.054 INFO:teuthology.orchestra.run.smithi149.stderr:instructed osd(s) 0 to deep-scrub 2023-09-18T20:44:44.065 INFO:tasks.ceph:Scrubbing osd.1 2023-09-18T20:44:44.066 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-18T20:44:44.210 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2023-09-18T20:44:44.210 INFO:teuthology.orchestra.run.smithi149.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-18T20:44:44.211 INFO:teuthology.orchestra.run.smithi149.stdout:} 2023-09-18T20:44:44.222 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-18T20:44:44.546 INFO:teuthology.orchestra.run.smithi149.stderr:instructed osd(s) 1 to deep-scrub 2023-09-18T20:44:44.557 INFO:tasks.ceph:Scrubbing osd.2 2023-09-18T20:44:44.558 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-18T20:44:44.702 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2023-09-18T20:44:44.702 INFO:teuthology.orchestra.run.smithi149.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-18T20:44:44.703 INFO:teuthology.orchestra.run.smithi149.stdout:} 2023-09-18T20:44:44.714 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-18T20:44:45.035 INFO:teuthology.orchestra.run.smithi149.stderr:instructed osd(s) 2 to deep-scrub 2023-09-18T20:44:45.046 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:44:45.367 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:44:45.369 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:44:45.381 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2023-09-18T20:44:43.724814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216918908,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":169166,"num_read_kb":672858,"num_write":169190,"num_write_kb":4772727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171646,"ondisk_log_size":171646,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9619768,"kb_used_data":8196832,"kb_used_omap":4,"kb_used_meta":1422907,"kb_avail":273495752,"statfs":{"total":289910292480,"available":280059650048,"internally_reserved":0,"allocated":8393555968,"data_stored":8401775777,"data_compressed":334018,"data_compressed_allocated":2260992,"data_compressed_original":11067392,"omap_allocated":4782,"internal_metadata":1457057106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":353,"num_read_kb":297,"num_write":5,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002492"},"pg_stats":[{"pgid":"3.1f","version":"31'5467","reported_seq":10804,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.490803+0000","last_change":"2023-09-18T20:40:40.833224+0000","last_active":"2023-09-18T20:43:57.490803+0000","last_peered":"2023-09-18T20:43:57.490803+0000","last_clean":"2023-09-18T20:43:57.490803+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:43:57.490803+0000","last_undegraded":"2023-09-18T20:43:57.490803+0000","last_fullsized":"2023-09-18T20:43:57.490803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:34:11.826626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4969","reported_seq":9780,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.504293+0000","last_change":"2023-09-18T20:40:40.786113+0000","last_active":"2023-09-18T20:43:57.504293+0000","last_peered":"2023-09-18T20:43:57.504293+0000","last_clean":"2023-09-18T20:43:57.504293+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:43:57.504293+0000","last_undegraded":"2023-09-18T20:43:57.504293+0000","last_fullsized":"2023-09-18T20:43:57.504293+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T20:55:47.617213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'5712","reported_seq":11417,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.449589+0000","last_change":"2023-09-18T20:40:40.785804+0000","last_active":"2023-09-18T20:43:57.449589+0000","last_peered":"2023-09-18T20:43:57.449589+0000","last_clean":"2023-09-18T20:43:57.449589+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:43:57.449589+0000","last_undegraded":"2023-09-18T20:43:57.449589+0000","last_fullsized":"2023-09-18T20:43:57.449589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:50:15.757762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614528,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5659,"num_write_kb":161724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5892","reported_seq":11756,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:33.200432+0000","last_change":"2023-09-18T20:40:40.790340+0000","last_active":"2023-09-18T20:44:33.200432+0000","last_peered":"2023-09-18T20:44:33.200432+0000","last_clean":"2023-09-18T20:44:33.200432+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:44:33.200432+0000","last_undegraded":"2023-09-18T20:44:33.200432+0000","last_fullsized":"2023-09-18T20:44:33.200432+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:25:56.139626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5838,"num_read_kb":23171,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5576","reported_seq":11077,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.480878+0000","last_change":"2023-09-18T20:40:40.832137+0000","last_active":"2023-09-18T20:43:57.480878+0000","last_peered":"2023-09-18T20:43:57.480878+0000","last_clean":"2023-09-18T20:43:57.480878+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:43:57.480878+0000","last_undegraded":"2023-09-18T20:43:57.480878+0000","last_fullsized":"2023-09-18T20:43:57.480878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:32:54.275405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"31'4802","reported_seq":9864,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:40.599111+0000","last_change":"2023-09-18T20:40:40.833200+0000","last_active":"2023-09-18T20:44:40.599111+0000","last_peered":"2023-09-18T20:44:40.599111+0000","last_clean":"2023-09-18T20:44:40.599111+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:44:40.599111+0000","last_undegraded":"2023-09-18T20:44:40.599111+0000","last_fullsized":"2023-09-18T20:44:40.599111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:06:48.959386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124866560,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5082,"num_read_kb":20361,"num_write":4767,"num_write_kb":138221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.888009+0000","last_change":"2023-09-18T20:39:18.754622+0000","last_active":"2023-09-18T20:43:27.888009+0000","last_peered":"2023-09-18T20:43:27.888009+0000","last_clean":"2023-09-18T20:43:27.888009+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:43:27.888009+0000","last_undegraded":"2023-09-18T20:43:27.888009+0000","last_fullsized":"2023-09-18T20:43:27.888009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:11:14.585598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887711+0000","last_change":"2023-09-18T20:39:18.755105+0000","last_active":"2023-09-18T20:43:27.887711+0000","last_peered":"2023-09-18T20:43:27.887711+0000","last_clean":"2023-09-18T20:43:27.887711+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:43:27.887711+0000","last_undegraded":"2023-09-18T20:43:27.887711+0000","last_fullsized":"2023-09-18T20:43:27.887711+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:09:50.066257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'4660","reported_seq":9362,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.431297+0000","last_change":"2023-09-18T20:40:40.786389+0000","last_active":"2023-09-18T20:43:57.431297+0000","last_peered":"2023-09-18T20:43:57.431297+0000","last_clean":"2023-09-18T20:43:57.431297+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:43:57.431297+0000","last_undegraded":"2023-09-18T20:43:57.431297+0000","last_fullsized":"2023-09-18T20:43:57.431297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:01:55.705787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997568,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4614,"num_write_kb":133028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'7540","reported_seq":15114,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.499606+0000","last_change":"2023-09-18T20:40:40.791026+0000","last_active":"2023-09-18T20:43:58.499606+0000","last_peered":"2023-09-18T20:43:58.499606+0000","last_clean":"2023-09-18T20:43:58.499606+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:43:58.499606+0000","last_undegraded":"2023-09-18T20:43:58.499606+0000","last_fullsized":"2023-09-18T20:43:58.499606+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:47:41.555603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.891231+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:43:27.891231+0000","last_peered":"2023-09-18T20:43:27.891231+0000","last_clean":"2023-09-18T20:43:27.891231+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:43:27.891231+0000","last_undegraded":"2023-09-18T20:43:27.891231+0000","last_fullsized":"2023-09-18T20:43:27.891231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:15:46.444320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6505","reported_seq":12883,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.464898+0000","last_change":"2023-09-18T20:40:40.831880+0000","last_active":"2023-09-18T20:43:57.464898+0000","last_peered":"2023-09-18T20:43:57.464898+0000","last_clean":"2023-09-18T20:43:57.464898+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:43:57.464898+0000","last_undegraded":"2023-09-18T20:43:57.464898+0000","last_fullsized":"2023-09-18T20:43:57.464898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:39:27.193036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.873392+0000","last_change":"2023-09-18T20:39:18.753469+0000","last_active":"2023-09-18T20:43:26.873392+0000","last_peered":"2023-09-18T20:43:26.873392+0000","last_clean":"2023-09-18T20:43:26.873392+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:43:26.873392+0000","last_undegraded":"2023-09-18T20:43:26.873392+0000","last_fullsized":"2023-09-18T20:43:26.873392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:22:56.897789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'4259","reported_seq":8454,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.452680+0000","last_change":"2023-09-18T20:40:40.790444+0000","last_active":"2023-09-18T20:43:57.452680+0000","last_peered":"2023-09-18T20:43:57.452680+0000","last_clean":"2023-09-18T20:43:57.452680+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:43:57.452680+0000","last_undegraded":"2023-09-18T20:43:57.452680+0000","last_fullsized":"2023-09-18T20:43:57.452680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:15:42.288917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":159,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.880206+0000","last_change":"2023-09-18T20:39:12.868453+0000","last_active":"2023-09-18T20:43:26.880206+0000","last_peered":"2023-09-18T20:43:26.880206+0000","last_clean":"2023-09-18T20:43:26.880206+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:43:26.880206+0000","last_undegraded":"2023-09-18T20:43:26.880206+0000","last_fullsized":"2023-09-18T20:43:26.880206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:11.726727+0000","last_clean_scrub_stamp":"2023-09-18T20:39:11.726727+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:32:10.120164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.887509+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:43:27.887509+0000","last_peered":"2023-09-18T20:43:27.887509+0000","last_clean":"2023-09-18T20:43:27.887509+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:43:27.887509+0000","last_undegraded":"2023-09-18T20:43:27.887509+0000","last_fullsized":"2023-09-18T20:43:27.887509+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:58:59.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'5837","reported_seq":11563,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.499918+0000","last_change":"2023-09-18T20:40:40.790900+0000","last_active":"2023-09-18T20:43:58.499918+0000","last_peered":"2023-09-18T20:43:58.499918+0000","last_clean":"2023-09-18T20:43:58.499918+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:43:58.499918+0000","last_undegraded":"2023-09-18T20:43:58.499918+0000","last_fullsized":"2023-09-18T20:43:58.499918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:17:11.468556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:26.875264+0000","last_change":"2023-09-18T20:39:18.753462+0000","last_active":"2023-09-18T20:43:26.875264+0000","last_peered":"2023-09-18T20:43:26.875264+0000","last_clean":"2023-09-18T20:43:26.875264+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:43:26.875264+0000","last_undegraded":"2023-09-18T20:43:26.875264+0000","last_fullsized":"2023-09-18T20:43:26.875264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:13:27.835062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"31'5805","reported_seq":11502,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.500020+0000","last_change":"2023-09-18T20:40:40.832355+0000","last_active":"2023-09-18T20:43:57.500020+0000","last_peered":"2023-09-18T20:43:57.500020+0000","last_clean":"2023-09-18T20:43:57.500020+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:43:57.500020+0000","last_undegraded":"2023-09-18T20:43:57.500020+0000","last_fullsized":"2023-09-18T20:43:57.500020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:44:19.351290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.147408+0000","last_change":"2023-09-18T20:39:18.753362+0000","last_active":"2023-09-18T20:43:27.147408+0000","last_peered":"2023-09-18T20:43:27.147408+0000","last_clean":"2023-09-18T20:43:27.147408+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:43:27.147408+0000","last_undegraded":"2023-09-18T20:43:27.147408+0000","last_fullsized":"2023-09-18T20:43:27.147408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:57:37.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6170","reported_seq":12274,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.430133+0000","last_change":"2023-09-18T20:40:40.786258+0000","last_active":"2023-09-18T20:43:57.430133+0000","last_peered":"2023-09-18T20:43:57.430133+0000","last_clean":"2023-09-18T20:43:57.430133+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:43:57.430133+0000","last_undegraded":"2023-09-18T20:43:57.430133+0000","last_fullsized":"2023-09-18T20:43:57.430133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:33:56.555234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:27.889508+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:43:27.889508+0000","last_peered":"2023-09-18T20:43:27.889508+0000","last_clean":"2023-09-18T20:43:27.889508+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:43:27.889508+0000","last_undegraded":"2023-09-18T20:43:27.889508+0000","last_fullsized":"2023-09-18T20:43:27.889508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:46:11.076182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"31'5238","reported_seq":10420,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486825+0000","last_change":"2023-09-18T20:40:40.785694+0000","last_active":"2023-09-18T20:43:57.486825+0000","last_peered":"2023-09-18T20:43:57.486825+0000","last_clean":"2023-09-18T20:43:57.486825+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:43:57.486825+0000","last_undegraded":"2023-09-18T20:43:57.486825+0000","last_fullsized":"2023-09-18T20:43:57.486825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:48:25.383550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"31'3820","reported_seq":7551,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:36.375135+0000","last_change":"2023-09-18T20:40:40.831884+0000","last_active":"2023-09-18T20:44:36.375135+0000","last_peered":"2023-09-18T20:44:36.375135+0000","last_clean":"2023-09-18T20:44:36.375135+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:44:36.375135+0000","last_undegraded":"2023-09-18T20:44:36.375135+0000","last_fullsized":"2023-09-18T20:44:36.375135+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:26:57.735585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3828,"num_read_kb":14459,"num_write":3566,"num_write_kb":100156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"31'7146","reported_seq":13960,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:40.645991+0000","last_change":"2023-09-18T20:40:40.832373+0000","last_active":"2023-09-18T20:44:40.645991+0000","last_peered":"2023-09-18T20:44:40.645991+0000","last_clean":"2023-09-18T20:44:40.645991+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:44:40.645991+0000","last_undegraded":"2023-09-18T20:44:40.645991+0000","last_fullsized":"2023-09-18T20:44:40.645991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:23:31.794226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":6998,"num_read_kb":27187,"num_write":6864,"num_write_kb":191656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'3581","reported_seq":7123,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.402162+0000","last_change":"2023-09-18T20:40:40.790576+0000","last_active":"2023-09-18T20:43:57.402162+0000","last_peered":"2023-09-18T20:43:57.402162+0000","last_clean":"2023-09-18T20:43:57.402162+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:43:57.402162+0000","last_undegraded":"2023-09-18T20:43:57.402162+0000","last_fullsized":"2023-09-18T20:43:57.402162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:13:44.471080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'4632","reported_seq":9022,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:41.498493+0000","last_change":"2023-09-18T20:40:40.832593+0000","last_active":"2023-09-18T20:44:41.498493+0000","last_peered":"2023-09-18T20:44:41.498493+0000","last_clean":"2023-09-18T20:44:41.498493+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:44:41.498493+0000","last_undegraded":"2023-09-18T20:44:41.498493+0000","last_fullsized":"2023-09-18T20:44:41.498493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:21:52.199914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108093440,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4486,"num_read_kb":17534,"num_write":4370,"num_write_kb":120492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"31'4114","reported_seq":8338,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.454474+0000","last_change":"2023-09-18T20:40:40.791131+0000","last_active":"2023-09-18T20:43:57.454474+0000","last_peered":"2023-09-18T20:43:57.454474+0000","last_clean":"2023-09-18T20:43:57.454474+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:43:57.454474+0000","last_undegraded":"2023-09-18T20:43:57.454474+0000","last_fullsized":"2023-09-18T20:43:57.454474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:05:41.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"31'6084","reported_seq":12190,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.488321+0000","last_change":"2023-09-18T20:40:40.786504+0000","last_active":"2023-09-18T20:43:57.488321+0000","last_peered":"2023-09-18T20:43:57.488321+0000","last_clean":"2023-09-18T20:43:57.488321+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:43:57.488321+0000","last_undegraded":"2023-09-18T20:43:57.488321+0000","last_fullsized":"2023-09-18T20:43:57.488321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:22:28.068618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'4159","reported_seq":8164,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:36.375127+0000","last_change":"2023-09-18T20:40:40.833389+0000","last_active":"2023-09-18T20:44:36.375127+0000","last_peered":"2023-09-18T20:44:36.375127+0000","last_clean":"2023-09-18T20:44:36.375127+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:44:36.375127+0000","last_undegraded":"2023-09-18T20:44:36.375127+0000","last_fullsized":"2023-09-18T20:44:36.375127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:27.656176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4108,"num_read_kb":15537,"num_write":3893,"num_write_kb":109673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"31'4527","reported_seq":9042,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.486241+0000","last_change":"2023-09-18T20:40:40.789756+0000","last_active":"2023-09-18T20:43:57.486241+0000","last_peered":"2023-09-18T20:43:57.486241+0000","last_clean":"2023-09-18T20:43:57.486241+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:43:57.486241+0000","last_undegraded":"2023-09-18T20:43:57.486241+0000","last_fullsized":"2023-09-18T20:43:57.486241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:33:32.442279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'6247","reported_seq":12323,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.495507+0000","last_change":"2023-09-18T20:40:40.785973+0000","last_active":"2023-09-18T20:43:58.495507+0000","last_peered":"2023-09-18T20:43:58.495507+0000","last_clean":"2023-09-18T20:43:58.495507+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:43:58.495507+0000","last_undegraded":"2023-09-18T20:43:58.495507+0000","last_fullsized":"2023-09-18T20:43:58.495507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:57:30.344645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'4971","reported_seq":9935,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:40.502564+0000","last_change":"2023-09-18T20:40:40.832115+0000","last_active":"2023-09-18T20:44:40.502564+0000","last_peered":"2023-09-18T20:44:40.502564+0000","last_clean":"2023-09-18T20:44:40.502564+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:44:40.502564+0000","last_undegraded":"2023-09-18T20:44:40.502564+0000","last_fullsized":"2023-09-18T20:44:40.502564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:29:42.639677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643058,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":4927,"num_read_kb":19684,"num_write":4934,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4581","reported_seq":9040,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.453882+0000","last_change":"2023-09-18T20:40:40.790674+0000","last_active":"2023-09-18T20:43:57.453882+0000","last_peered":"2023-09-18T20:43:57.453882+0000","last_clean":"2023-09-18T20:43:57.453882+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:43:57.453882+0000","last_undegraded":"2023-09-18T20:43:57.453882+0000","last_fullsized":"2023-09-18T20:43:57.453882+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:32:31.952190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"31'4431","reported_seq":8836,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.465603+0000","last_change":"2023-09-18T20:40:40.790794+0000","last_active":"2023-09-18T20:43:57.465603+0000","last_peered":"2023-09-18T20:43:57.465603+0000","last_clean":"2023-09-18T20:43:57.465603+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:43:57.465603+0000","last_undegraded":"2023-09-18T20:43:57.465603+0000","last_fullsized":"2023-09-18T20:43:57.465603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:27:46.524729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'5099","reported_seq":10331,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.457884+0000","last_change":"2023-09-18T20:40:40.832637+0000","last_active":"2023-09-18T20:43:57.457884+0000","last_peered":"2023-09-18T20:43:57.457884+0000","last_clean":"2023-09-18T20:43:57.457884+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:43:57.457884+0000","last_undegraded":"2023-09-18T20:43:57.457884+0000","last_fullsized":"2023-09-18T20:43:57.457884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:22:01.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"31'6089","reported_seq":11936,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.510475+0000","last_change":"2023-09-18T20:40:40.786619+0000","last_active":"2023-09-18T20:43:57.510475+0000","last_peered":"2023-09-18T20:43:57.510475+0000","last_clean":"2023-09-18T20:43:57.510475+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:43:57.510475+0000","last_undegraded":"2023-09-18T20:43:57.510475+0000","last_fullsized":"2023-09-18T20:43:57.510475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:30:31.491319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'6927","reported_seq":13657,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:40.642266+0000","last_change":"2023-09-18T20:40:40.833412+0000","last_active":"2023-09-18T20:44:40.642266+0000","last_peered":"2023-09-18T20:44:40.642266+0000","last_clean":"2023-09-18T20:44:40.642266+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:44:40.642266+0000","last_undegraded":"2023-09-18T20:44:40.642266+0000","last_fullsized":"2023-09-18T20:44:40.642266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:40:52.246018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6837,"num_read_kb":26409,"num_write":6656,"num_write_kb":186153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6672","reported_seq":13399,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:57.450693+0000","last_change":"2023-09-18T20:40:40.785488+0000","last_active":"2023-09-18T20:43:57.450693+0000","last_peered":"2023-09-18T20:43:57.450693+0000","last_clean":"2023-09-18T20:43:57.450693+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:43:57.450693+0000","last_undegraded":"2023-09-18T20:43:57.450693+0000","last_fullsized":"2023-09-18T20:43:57.450693+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:43:31.948097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390656,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6622,"num_write_kb":190144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'5371","reported_seq":10457,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:43:58.504285+0000","last_change":"2023-09-18T20:40:40.789720+0000","last_active":"2023-09-18T20:43:58.504285+0000","last_peered":"2023-09-18T20:43:58.504285+0000","last_clean":"2023-09-18T20:43:58.504285+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:43:58.504285+0000","last_undegraded":"2023-09-18T20:43:58.504285+0000","last_fullsized":"2023-09-18T20:43:58.504285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:56:59.436258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"31'4666","reported_seq":9363,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:40.597892+0000","last_change":"2023-09-18T20:40:40.790245+0000","last_active":"2023-09-18T20:44:40.597892+0000","last_peered":"2023-09-18T20:44:40.597892+0000","last_clean":"2023-09-18T20:44:40.597892+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:44:40.597892+0000","last_undegraded":"2023-09-18T20:44:40.597892+0000","last_fullsized":"2023-09-18T20:44:40.597892+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:34:55.473325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4660,"num_read_kb":18900,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216328540,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":169060,"num_read_kb":672767,"num_write":169053,"num_write_kb":4770833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392654848,"data_stored":8400298680,"data_compressed":322494,"data_compressed_allocated":2187264,"data_compressed_original":9920512,"omap_allocated":0,"internal_metadata":0},"log_size":171549,"ondisk_log_size":171549,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738437,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2816096,"kb_used_data":2397784,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91555744,"statfs":{"total":96636764160,"available":93753081856,"internally_reserved":0,"allocated":2455330816,"data_stored":2456646266,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0660000000000001,"15min":0.55100000000000005},"min":{"1min":0.16700000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":3.4380000000000002,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.04,"15min":0.54900000000000004},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.5730000000000004,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.3260000000000001,"15min":0.64400000000000002},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.629,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.3160000000000001,"15min":0.65200000000000002},"min":{"1min":0.27100000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.6289999999999996,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2974200,"kb_used_data":2532272,"kb_used_omap":1,"kb_used_meta":441918,"kb_avail":91397640,"statfs":{"total":96636764160,"available":93591183360,"internally_reserved":0,"allocated":2593046528,"data_stored":2595779834,"data_compressed":59035,"data_compressed_allocated":430080,"data_compressed_original":3358720,"omap_allocated":1594,"internal_metadata":452524486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.46300000000000002,"15min":0.34699999999999998},"min":{"1min":0.18099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.59399999999999997,"15min":0.40500000000000003},"min":{"1min":0.189,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.0049999999999999,"5min":3.585,"15min":3.585},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.39200000000000002},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5489999999999999,"5min":12.382,"15min":12.382},"last":0.251},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.53400000000000003,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738437,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3829472,"kb_used_data":3266776,"kb_used_omap":1,"kb_used_meta":562686,"kb_avail":90542368,"statfs":{"total":96636764160,"available":92715384832,"internally_reserved":0,"allocated":3345178624,"data_stored":3349349677,"data_compressed":165270,"data_compressed_allocated":1118208,"data_compressed_original":5496832,"omap_allocated":1594,"internal_metadata":576190918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80900000000000005,"15min":0.45700000000000002},"min":{"1min":0.23799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":5.6150000000000002,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.9079999999999999,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.94199999999999995,"15min":0.503},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.4710000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.1379999999999999,"15min":0.58399999999999996},"min":{"1min":0.26300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":5.5049999999999999,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592727040,"data_stored":2595090679,"data_compressed":53273,"data_compressed_allocated":393216,"data_compressed_original":2785280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344859136,"data_stored":3348660522,"data_compressed":159508,"data_compressed_allocated":1081344,"data_compressed_original":4923392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455068672,"data_stored":2456547479,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:44:45.385 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.385 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.386 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.386 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.386 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.387 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.387 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.388 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.388 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.388 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.389 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.389 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.389 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.390 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.390 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-18T20:39:11.726727+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=11, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.390 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.391 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.391 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.391 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.392 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.392 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.392 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.393 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.393 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.393 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.394 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.394 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.394 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.395 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.395 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.395 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.396 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.396 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.396 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.397 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.397 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.397 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.398 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.398 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.398 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.399 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:44:45.399 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-18T20:45:05.400 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:45:05.767 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:45:05.770 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:45:05.784 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2023-09-18T20:45:05.729710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216919158,"num_objects":1061,"num_object_clones":0,"num_object_copies":2122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":169576,"num_read_kb":673208,"num_write":169211,"num_write_kb":4772751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171850,"ondisk_log_size":171850,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9619844,"kb_used_data":8196896,"kb_used_omap":4,"kb_used_meta":1422907,"kb_avail":273495676,"statfs":{"total":289910292480,"available":280059572224,"internally_reserved":0,"allocated":8393621504,"data_stored":8401788289,"data_compressed":334018,"data_compressed_allocated":2260992,"data_compressed_original":11067392,"omap_allocated":4782,"internal_metadata":1457057106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":250,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":410,"num_read_kb":350,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002143"},"pg_stats":[{"pgid":"3.1f","version":"31'5467","reported_seq":10808,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787650+0000","last_change":"2023-09-18T20:40:40.833224+0000","last_active":"2023-09-18T20:45:00.787650+0000","last_peered":"2023-09-18T20:45:00.787650+0000","last_clean":"2023-09-18T20:45:00.787650+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:45:00.787650+0000","last_undegraded":"2023-09-18T20:45:00.787650+0000","last_fullsized":"2023-09-18T20:45:00.787650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4969","reported_seq":9783,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783668+0000","last_change":"2023-09-18T20:40:40.786113+0000","last_active":"2023-09-18T20:45:00.783668+0000","last_peered":"2023-09-18T20:45:00.783668+0000","last_clean":"2023-09-18T20:45:00.783668+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:45:00.783668+0000","last_undegraded":"2023-09-18T20:45:00.783668+0000","last_fullsized":"2023-09-18T20:45:00.783668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"32'5713","reported_seq":11421,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783683+0000","last_change":"2023-09-18T20:40:40.785804+0000","last_active":"2023-09-18T20:45:00.783683+0000","last_peered":"2023-09-18T20:45:00.783683+0000","last_clean":"2023-09-18T20:45:00.783683+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:45:00.783683+0000","last_undegraded":"2023-09-18T20:45:00.783683+0000","last_fullsized":"2023-09-18T20:45:00.783683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5713,"log_dups_size":0,"ondisk_log_size":5713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614578,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5660,"num_write_kb":161725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5892","reported_seq":11769,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.798039+0000","last_change":"2023-09-18T20:40:40.790340+0000","last_active":"2023-09-18T20:45:00.798039+0000","last_peered":"2023-09-18T20:45:00.798039+0000","last_clean":"2023-09-18T20:45:00.798039+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:45:00.798039+0000","last_undegraded":"2023-09-18T20:45:00.798039+0000","last_fullsized":"2023-09-18T20:45:00.798039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5848,"num_read_kb":23181,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5576","reported_seq":11080,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787886+0000","last_change":"2023-09-18T20:40:40.832137+0000","last_active":"2023-09-18T20:45:00.787886+0000","last_peered":"2023-09-18T20:45:00.787886+0000","last_clean":"2023-09-18T20:45:00.787886+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:45:00.787886+0000","last_undegraded":"2023-09-18T20:45:00.787886+0000","last_fullsized":"2023-09-18T20:45:00.787886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"31'4803","reported_seq":9889,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787740+0000","last_change":"2023-09-18T20:40:40.833200+0000","last_active":"2023-09-18T20:45:00.787740+0000","last_peered":"2023-09-18T20:45:00.787740+0000","last_clean":"2023-09-18T20:45:00.787740+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:45:00.787740+0000","last_undegraded":"2023-09-18T20:45:00.787740+0000","last_fullsized":"2023-09-18T20:45:00.787740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124866560,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5102,"num_read_kb":20381,"num_write":4768,"num_write_kb":138225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.203501+0000","last_change":"2023-09-18T20:45:00.203501+0000","last_active":"2023-09-18T20:45:00.203501+0000","last_peered":"2023-09-18T20:45:00.203501+0000","last_clean":"2023-09-18T20:45:00.203501+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:45:00.203501+0000","last_undegraded":"2023-09-18T20:45:00.203501+0000","last_fullsized":"2023-09-18T20:45:00.203501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:00.203367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:00.203367+0000","last_clean_scrub_stamp":"2023-09-18T20:45:00.203367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:09:19.993921+0000","scrub_duration":0.0012992380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:51.192940+0000","last_change":"2023-09-18T20:44:51.192940+0000","last_active":"2023-09-18T20:44:51.192940+0000","last_peered":"2023-09-18T20:44:51.192940+0000","last_clean":"2023-09-18T20:44:51.192940+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:44:51.192940+0000","last_undegraded":"2023-09-18T20:44:51.192940+0000","last_fullsized":"2023-09-18T20:44:51.192940+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:51.192786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:51.192786+0000","last_clean_scrub_stamp":"2023-09-18T20:44:51.192786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:12:10.502907+0000","scrub_duration":0.001553362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"32'4694","reported_seq":9409,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.809715+0000","last_change":"2023-09-18T20:44:53.516136+0000","last_active":"2023-09-18T20:45:00.809715+0000","last_peered":"2023-09-18T20:45:00.809715+0000","last_clean":"2023-09-18T20:45:00.809715+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:45:00.809715+0000","last_undegraded":"2023-09-18T20:45:00.809715+0000","last_fullsized":"2023-09-18T20:45:00.809715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4692","last_scrub_stamp":"2023-09-18T20:44:53.515992+0000","last_deep_scrub":"31'4692","last_deep_scrub_stamp":"2023-09-18T20:44:53.515992+0000","last_clean_scrub_stamp":"2023-09-18T20:44:53.515992+0000","objects_scrubbed":32,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:17:08.007446+0000","scrub_duration":0.339401179,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997668,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4616,"num_write_kb":133030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'7540","reported_seq":15118,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797340+0000","last_change":"2023-09-18T20:40:40.791026+0000","last_active":"2023-09-18T20:45:00.797340+0000","last_peered":"2023-09-18T20:45:00.797340+0000","last_clean":"2023-09-18T20:45:00.797340+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:45:00.797340+0000","last_undegraded":"2023-09-18T20:45:00.797340+0000","last_fullsized":"2023-09-18T20:45:00.797340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:44.543587+0000","last_change":"2023-09-18T20:39:18.754334+0000","last_active":"2023-09-18T20:44:44.543587+0000","last_peered":"2023-09-18T20:44:44.543587+0000","last_clean":"2023-09-18T20:44:44.543587+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:44:44.543587+0000","last_undegraded":"2023-09-18T20:44:44.543587+0000","last_fullsized":"2023-09-18T20:44:44.543587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6544","reported_seq":12935,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787507+0000","last_change":"2023-09-18T20:44:56.214593+0000","last_active":"2023-09-18T20:45:00.787507+0000","last_peered":"2023-09-18T20:45:00.787507+0000","last_clean":"2023-09-18T20:45:00.787507+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:45:00.787507+0000","last_undegraded":"2023-09-18T20:45:00.787507+0000","last_fullsized":"2023-09-18T20:45:00.787507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6544","last_scrub_stamp":"2023-09-18T20:44:56.214397+0000","last_deep_scrub":"31'6544","last_deep_scrub_stamp":"2023-09-18T20:44:56.214397+0000","last_clean_scrub_stamp":"2023-09-18T20:44:56.214397+0000","objects_scrubbed":39,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:25:51.626829+0000","scrub_duration":0.4789736,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:47.320025+0000","last_change":"2023-09-18T20:44:47.320025+0000","last_active":"2023-09-18T20:44:47.320025+0000","last_peered":"2023-09-18T20:44:47.320025+0000","last_clean":"2023-09-18T20:44:47.320025+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:44:47.320025+0000","last_undegraded":"2023-09-18T20:44:47.320025+0000","last_fullsized":"2023-09-18T20:44:47.320025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:47.319876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:47.319876+0000","last_clean_scrub_stamp":"2023-09-18T20:44:47.319876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:33:13.500197+0000","scrub_duration":0.001796179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'4259","reported_seq":8458,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796900+0000","last_change":"2023-09-18T20:40:40.790444+0000","last_active":"2023-09-18T20:45:00.796900+0000","last_peered":"2023-09-18T20:45:00.796900+0000","last_clean":"2023-09-18T20:45:00.796900+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:45:00.796900+0000","last_undegraded":"2023-09-18T20:45:00.796900+0000","last_fullsized":"2023-09-18T20:45:00.796900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'99","reported_seq":171,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:50.174363+0000","last_change":"2023-09-18T20:44:50.174363+0000","last_active":"2023-09-18T20:44:50.174363+0000","last_peered":"2023-09-18T20:44:50.174363+0000","last_clean":"2023-09-18T20:44:50.174363+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:44:50.174363+0000","last_undegraded":"2023-09-18T20:44:50.174363+0000","last_fullsized":"2023-09-18T20:44:50.174363+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'99","last_scrub_stamp":"2023-09-18T20:44:50.174236+0000","last_deep_scrub":"31'99","last_deep_scrub_stamp":"2023-09-18T20:44:50.174236+0000","last_clean_scrub_stamp":"2023-09-18T20:44:50.174236+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:53:49.471937+0000","scrub_duration":0.0039956039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:44.543649+0000","last_change":"2023-09-18T20:39:18.753326+0000","last_active":"2023-09-18T20:44:44.543649+0000","last_peered":"2023-09-18T20:44:44.543649+0000","last_clean":"2023-09-18T20:44:44.543649+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:44:44.543649+0000","last_undegraded":"2023-09-18T20:44:44.543649+0000","last_fullsized":"2023-09-18T20:44:44.543649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'5837","reported_seq":11566,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796748+0000","last_change":"2023-09-18T20:40:40.790900+0000","last_active":"2023-09-18T20:45:00.796748+0000","last_peered":"2023-09-18T20:45:00.796748+0000","last_clean":"2023-09-18T20:45:00.796748+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:45:00.796748+0000","last_undegraded":"2023-09-18T20:45:00.796748+0000","last_fullsized":"2023-09-18T20:45:00.796748+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:50.240541+0000","last_change":"2023-09-18T20:44:50.240541+0000","last_active":"2023-09-18T20:44:50.240541+0000","last_peered":"2023-09-18T20:44:50.240541+0000","last_clean":"2023-09-18T20:44:50.240541+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:44:50.240541+0000","last_undegraded":"2023-09-18T20:44:50.240541+0000","last_fullsized":"2023-09-18T20:44:50.240541+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:50.240423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:50.240423+0000","last_clean_scrub_stamp":"2023-09-18T20:44:50.240423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:40:37.117429+0000","scrub_duration":0.001250791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"31'5805","reported_seq":11506,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787794+0000","last_change":"2023-09-18T20:40:40.832355+0000","last_active":"2023-09-18T20:45:00.787794+0000","last_peered":"2023-09-18T20:45:00.787794+0000","last_clean":"2023-09-18T20:45:00.787794+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:45:00.787794+0000","last_undegraded":"2023-09-18T20:45:00.787794+0000","last_fullsized":"2023-09-18T20:45:00.787794+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:49.758943+0000","last_change":"2023-09-18T20:44:49.758943+0000","last_active":"2023-09-18T20:44:49.758943+0000","last_peered":"2023-09-18T20:44:49.758943+0000","last_clean":"2023-09-18T20:44:49.758943+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:44:49.758943+0000","last_undegraded":"2023-09-18T20:44:49.758943+0000","last_fullsized":"2023-09-18T20:44:49.758943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:49.758824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:49.758824+0000","last_clean_scrub_stamp":"2023-09-18T20:44:49.758824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:13:52.512458+0000","scrub_duration":0.0012109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6207","reported_seq":12324,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783860+0000","last_change":"2023-09-18T20:44:55.635468+0000","last_active":"2023-09-18T20:45:00.783860+0000","last_peered":"2023-09-18T20:45:00.783860+0000","last_clean":"2023-09-18T20:45:00.783860+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:45:00.783860+0000","last_undegraded":"2023-09-18T20:45:00.783860+0000","last_fullsized":"2023-09-18T20:45:00.783860+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6207","last_scrub_stamp":"2023-09-18T20:44:55.635254+0000","last_deep_scrub":"31'6207","last_deep_scrub_stamp":"2023-09-18T20:44:55.635254+0000","last_clean_scrub_stamp":"2023-09-18T20:44:55.635254+0000","objects_scrubbed":37,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:04:59.882463+0000","scrub_duration":0.45434064299999999,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:44.543597+0000","last_change":"2023-09-18T20:39:18.754465+0000","last_active":"2023-09-18T20:44:44.543597+0000","last_peered":"2023-09-18T20:44:44.543597+0000","last_clean":"2023-09-18T20:44:44.543597+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:44:44.543597+0000","last_undegraded":"2023-09-18T20:44:44.543597+0000","last_fullsized":"2023-09-18T20:44:44.543597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:39:17.741265+0000","last_clean_scrub_stamp":"2023-09-18T20:39:17.741265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"31'5238","reported_seq":10424,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783515+0000","last_change":"2023-09-18T20:40:40.785694+0000","last_active":"2023-09-18T20:45:00.783515+0000","last_peered":"2023-09-18T20:45:00.783515+0000","last_clean":"2023-09-18T20:45:00.783515+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:45:00.783515+0000","last_undegraded":"2023-09-18T20:45:00.783515+0000","last_fullsized":"2023-09-18T20:45:00.783515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"32'3824","reported_seq":7590,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:01.975297+0000","last_change":"2023-09-18T20:40:40.831884+0000","last_active":"2023-09-18T20:45:01.975297+0000","last_peered":"2023-09-18T20:45:01.975297+0000","last_clean":"2023-09-18T20:45:01.975297+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:45:01.975297+0000","last_undegraded":"2023-09-18T20:45:01.975297+0000","last_fullsized":"2023-09-18T20:45:01.975297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3902,"num_read_kb":14521,"num_write":3569,"num_write_kb":100159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"32'7150","reported_seq":13999,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:01.975116+0000","last_change":"2023-09-18T20:40:40.832373+0000","last_active":"2023-09-18T20:45:01.975116+0000","last_peered":"2023-09-18T20:45:01.975116+0000","last_clean":"2023-09-18T20:45:01.975116+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:45:01.975116+0000","last_undegraded":"2023-09-18T20:45:01.975116+0000","last_fullsized":"2023-09-18T20:45:01.975116+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":7072,"num_read_kb":27249,"num_write":6867,"num_write_kb":191659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'3581","reported_seq":7126,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797111+0000","last_change":"2023-09-18T20:40:40.790576+0000","last_active":"2023-09-18T20:45:00.797111+0000","last_peered":"2023-09-18T20:45:00.797111+0000","last_clean":"2023-09-18T20:45:00.797111+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:45:00.797111+0000","last_undegraded":"2023-09-18T20:45:00.797111+0000","last_fullsized":"2023-09-18T20:45:00.797111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"32'4636","reported_seq":9060,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.824185+0000","last_change":"2023-09-18T20:40:40.832593+0000","last_active":"2023-09-18T20:45:00.824185+0000","last_peered":"2023-09-18T20:45:00.824185+0000","last_clean":"2023-09-18T20:45:00.824185+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:45:00.824185+0000","last_undegraded":"2023-09-18T20:45:00.824185+0000","last_fullsized":"2023-09-18T20:45:00.824185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108093440,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4560,"num_read_kb":17596,"num_write":4373,"num_write_kb":120495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"31'4114","reported_seq":8341,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797249+0000","last_change":"2023-09-18T20:40:40.791131+0000","last_active":"2023-09-18T20:45:00.797249+0000","last_peered":"2023-09-18T20:45:00.797249+0000","last_clean":"2023-09-18T20:45:00.797249+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:45:00.797249+0000","last_undegraded":"2023-09-18T20:45:00.797249+0000","last_fullsized":"2023-09-18T20:45:00.797249+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"31'6084","reported_seq":12193,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783808+0000","last_change":"2023-09-18T20:40:40.786504+0000","last_active":"2023-09-18T20:45:00.783808+0000","last_peered":"2023-09-18T20:45:00.783808+0000","last_clean":"2023-09-18T20:45:00.783808+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:45:00.783808+0000","last_undegraded":"2023-09-18T20:45:00.783808+0000","last_fullsized":"2023-09-18T20:45:00.783808+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"32'4163","reported_seq":8202,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.820236+0000","last_change":"2023-09-18T20:40:40.833389+0000","last_active":"2023-09-18T20:45:00.820236+0000","last_peered":"2023-09-18T20:45:00.820236+0000","last_clean":"2023-09-18T20:45:00.820236+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:45:00.820236+0000","last_undegraded":"2023-09-18T20:45:00.820236+0000","last_fullsized":"2023-09-18T20:45:00.820236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4182,"num_read_kb":15599,"num_write":3896,"num_write_kb":109676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"31'4527","reported_seq":9045,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796663+0000","last_change":"2023-09-18T20:40:40.789756+0000","last_active":"2023-09-18T20:45:00.796663+0000","last_peered":"2023-09-18T20:45:00.796663+0000","last_clean":"2023-09-18T20:45:00.796663+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:45:00.796663+0000","last_undegraded":"2023-09-18T20:45:00.796663+0000","last_fullsized":"2023-09-18T20:45:00.796663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'6247","reported_seq":12326,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783751+0000","last_change":"2023-09-18T20:40:40.785973+0000","last_active":"2023-09-18T20:45:00.783751+0000","last_peered":"2023-09-18T20:45:00.783751+0000","last_clean":"2023-09-18T20:45:00.783751+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:45:00.783751+0000","last_undegraded":"2023-09-18T20:45:00.783751+0000","last_fullsized":"2023-09-18T20:45:00.783751+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"32'4972","reported_seq":9939,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787408+0000","last_change":"2023-09-18T20:40:40.832115+0000","last_active":"2023-09-18T20:45:00.787408+0000","last_peered":"2023-09-18T20:45:00.787408+0000","last_clean":"2023-09-18T20:45:00.787408+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:45:00.787408+0000","last_undegraded":"2023-09-18T20:45:00.787408+0000","last_fullsized":"2023-09-18T20:45:00.787408+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643108,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4927,"num_read_kb":19684,"num_write":4935,"num_write_kb":138349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4608","reported_seq":9079,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797005+0000","last_change":"2023-09-18T20:44:49.462443+0000","last_active":"2023-09-18T20:45:00.797005+0000","last_peered":"2023-09-18T20:45:00.797005+0000","last_clean":"2023-09-18T20:45:00.797005+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:45:00.797005+0000","last_undegraded":"2023-09-18T20:45:00.797005+0000","last_fullsized":"2023-09-18T20:45:00.797005+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4608","last_scrub_stamp":"2023-09-18T20:44:49.462306+0000","last_deep_scrub":"31'4608","last_deep_scrub_stamp":"2023-09-18T20:44:49.462306+0000","last_clean_scrub_stamp":"2023-09-18T20:44:49.462306+0000","objects_scrubbed":27,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:54:51.936441+0000","scrub_duration":0.337859574,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"31'4431","reported_seq":8839,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797160+0000","last_change":"2023-09-18T20:40:40.790794+0000","last_active":"2023-09-18T20:45:00.797160+0000","last_peered":"2023-09-18T20:45:00.797160+0000","last_clean":"2023-09-18T20:45:00.797160+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:45:00.797160+0000","last_undegraded":"2023-09-18T20:45:00.797160+0000","last_fullsized":"2023-09-18T20:45:00.797160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'5099","reported_seq":10334,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787797+0000","last_change":"2023-09-18T20:40:40.832637+0000","last_active":"2023-09-18T20:45:00.787797+0000","last_peered":"2023-09-18T20:45:00.787797+0000","last_clean":"2023-09-18T20:45:00.787797+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:45:00.787797+0000","last_undegraded":"2023-09-18T20:45:00.787797+0000","last_fullsized":"2023-09-18T20:45:00.787797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"31'6089","reported_seq":11939,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783936+0000","last_change":"2023-09-18T20:40:40.786619+0000","last_active":"2023-09-18T20:45:00.783936+0000","last_peered":"2023-09-18T20:45:00.783936+0000","last_clean":"2023-09-18T20:45:00.783936+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:45:00.783936+0000","last_undegraded":"2023-09-18T20:45:00.783936+0000","last_fullsized":"2023-09-18T20:45:00.783936+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"32'6931","reported_seq":13695,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.826779+0000","last_change":"2023-09-18T20:40:40.833412+0000","last_active":"2023-09-18T20:45:00.826779+0000","last_peered":"2023-09-18T20:45:00.826779+0000","last_clean":"2023-09-18T20:45:00.826779+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:45:00.826779+0000","last_undegraded":"2023-09-18T20:45:00.826779+0000","last_fullsized":"2023-09-18T20:45:00.826779+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6911,"num_read_kb":26471,"num_write":6659,"num_write_kb":186156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"32'6714","reported_seq":13453,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783375+0000","last_change":"2023-09-18T20:44:52.726361+0000","last_active":"2023-09-18T20:45:00.783375+0000","last_peered":"2023-09-18T20:45:00.783375+0000","last_clean":"2023-09-18T20:45:00.783375+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:45:00.783375+0000","last_undegraded":"2023-09-18T20:45:00.783375+0000","last_fullsized":"2023-09-18T20:45:00.783375+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6713","last_scrub_stamp":"2023-09-18T20:44:52.726150+0000","last_deep_scrub":"31'6713","last_deep_scrub_stamp":"2023-09-18T20:44:52.726150+0000","last_clean_scrub_stamp":"2023-09-18T20:44:52.726150+0000","objects_scrubbed":41,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:53:09.298051+0000","scrub_duration":0.52103732999999997,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390706,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6623,"num_write_kb":190145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'5371","reported_seq":10460,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796939+0000","last_change":"2023-09-18T20:40:40.789720+0000","last_active":"2023-09-18T20:45:00.796939+0000","last_peered":"2023-09-18T20:45:00.796939+0000","last_clean":"2023-09-18T20:45:00.796939+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:45:00.796939+0000","last_undegraded":"2023-09-18T20:45:00.796939+0000","last_fullsized":"2023-09-18T20:45:00.796939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"31'4666","reported_seq":9376,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.821187+0000","last_change":"2023-09-18T20:40:40.790245+0000","last_active":"2023-09-18T20:45:00.821187+0000","last_peered":"2023-09-18T20:45:00.821187+0000","last_clean":"2023-09-18T20:45:00.821187+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:45:00.821187+0000","last_undegraded":"2023-09-18T20:45:00.821187+0000","last_fullsized":"2023-09-18T20:45:00.821187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4670,"num_read_kb":18910,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216328790,"num_objects":1059,"num_object_clones":0,"num_object_copies":2118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":169470,"num_read_kb":673117,"num_write":169074,"num_write_kb":4770857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392695808,"data_stored":8400299180,"data_compressed":322494,"data_compressed_allocated":2187264,"data_compressed_original":9920512,"omap_allocated":0,"internal_metadata":0},"log_size":171751,"ondisk_log_size":171751,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738441,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2816124,"kb_used_data":2397808,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91555716,"statfs":{"total":96636764160,"available":93753053184,"internally_reserved":0,"allocated":2455355392,"data_stored":2456650470,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0660000000000001,"15min":0.55100000000000005},"min":{"1min":0.16700000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":3.4380000000000002,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.208,"5min":1.04,"15min":0.54900000000000004},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.5730000000000004,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.3260000000000001,"15min":0.64400000000000002},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.629,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.3160000000000001,"15min":0.65200000000000002},"min":{"1min":0.27100000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.6289999999999996,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.434}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2974224,"kb_used_data":2532292,"kb_used_omap":1,"kb_used_meta":441918,"kb_avail":91397616,"statfs":{"total":96636764160,"available":93591158784,"internally_reserved":0,"allocated":2593067008,"data_stored":2595783988,"data_compressed":59035,"data_compressed_allocated":430080,"data_compressed_original":3358720,"omap_allocated":1594,"internal_metadata":452524486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.46300000000000002,"15min":0.34699999999999998},"min":{"1min":0.18099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.254},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.59399999999999997,"15min":0.40500000000000003},"min":{"1min":0.189,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.0049999999999999,"5min":3.585,"15min":3.585},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.39200000000000002},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5489999999999999,"5min":12.382,"15min":12.382},"last":0.312},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.53400000000000003,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":0.33000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3829496,"kb_used_data":3266796,"kb_used_omap":1,"kb_used_meta":562686,"kb_avail":90542344,"statfs":{"total":96636764160,"available":92715360256,"internally_reserved":0,"allocated":3345199104,"data_stored":3349353831,"data_compressed":165270,"data_compressed_allocated":1118208,"data_compressed_original":5496832,"omap_allocated":1594,"internal_metadata":576190918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80900000000000005,"15min":0.45700000000000002},"min":{"1min":0.23799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":5.6150000000000002,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.9079999999999999,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.94199999999999995,"15min":0.503},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.4710000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.1379999999999999,"15min":0.58399999999999996},"min":{"1min":0.26300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":5.5049999999999999,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592739328,"data_stored":2595090829,"data_compressed":53273,"data_compressed_allocated":393216,"data_compressed_original":2785280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344871424,"data_stored":3348660672,"data_compressed":159508,"data_compressed_allocated":1081344,"data_compressed_original":4923392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455085056,"data_stored":2456547679,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:45:05.787 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.788 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.788 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.788 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.789 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.789 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.789 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.790 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.790 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.791 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.791 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.792 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.792 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-18T20:39:17.741265+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=39, tm_sec=17, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.792 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.793 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.793 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.794 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.794 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.794 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.795 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.795 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.796 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.796 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.796 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.797 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.797 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.797 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.798 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.798 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.799 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:05.799 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-18T20:45:25.800 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:45:26.131 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:45:26.132 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:45:26.144 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2023-09-18T20:45:25.733956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216919158,"num_objects":1061,"num_object_clones":0,"num_object_copies":2122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":169576,"num_read_kb":673208,"num_write":169211,"num_write_kb":4772751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2446,"num_omap_keys":58,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172342,"ondisk_log_size":172342,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9623992,"kb_used_data":8196896,"kb_used_omap":4,"kb_used_meta":1427067,"kb_avail":273491528,"statfs":{"total":289910292480,"available":280055324672,"internally_reserved":0,"allocated":8393621504,"data_stored":8401788289,"data_compressed":334018,"data_compressed_allocated":2260992,"data_compressed_original":11067392,"omap_allocated":4782,"internal_metadata":1461316946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1431,"num_omap_keys":38,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002626"},"pg_stats":[{"pgid":"3.1f","version":"31'5467","reported_seq":10808,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787650+0000","last_change":"2023-09-18T20:40:40.833224+0000","last_active":"2023-09-18T20:45:00.787650+0000","last_peered":"2023-09-18T20:45:00.787650+0000","last_clean":"2023-09-18T20:45:00.787650+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:45:00.787650+0000","last_undegraded":"2023-09-18T20:45:00.787650+0000","last_fullsized":"2023-09-18T20:45:00.787650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"32'4999","reported_seq":9822,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:05.570688+0000","last_change":"2023-09-18T20:45:05.570748+0000","last_active":"2023-09-18T20:45:05.570688+0000","last_peered":"2023-09-18T20:45:05.570688+0000","last_clean":"2023-09-18T20:45:05.570688+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:45:05.570688+0000","last_undegraded":"2023-09-18T20:45:05.570688+0000","last_fullsized":"2023-09-18T20:45:05.570688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4999","last_scrub_stamp":"2023-09-18T20:45:05.570612+0000","last_deep_scrub":"32'4999","last_deep_scrub_stamp":"2023-09-18T20:45:05.570612+0000","last_clean_scrub_stamp":"2023-09-18T20:45:05.570612+0000","objects_scrubbed":30,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:08:24.598326+0000","scrub_duration":0.41260589600000003,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"32'5748","reported_seq":11465,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:07.609695+0000","last_change":"2023-09-18T20:45:07.609758+0000","last_active":"2023-09-18T20:45:07.609695+0000","last_peered":"2023-09-18T20:45:07.609695+0000","last_clean":"2023-09-18T20:45:07.609695+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:45:07.609695+0000","last_undegraded":"2023-09-18T20:45:07.609695+0000","last_fullsized":"2023-09-18T20:45:07.609695+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5748","last_scrub_stamp":"2023-09-18T20:45:07.609608+0000","last_deep_scrub":"32'5748","last_deep_scrub_stamp":"2023-09-18T20:45:07.609608+0000","last_clean_scrub_stamp":"2023-09-18T20:45:07.609608+0000","objects_scrubbed":36,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:43:28.999233+0000","scrub_duration":0.43395783999999998,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614578,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5660,"num_write_kb":161725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"32'5928","reported_seq":11814,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:20.625971+0000","last_change":"2023-09-18T20:45:20.626030+0000","last_active":"2023-09-18T20:45:20.625971+0000","last_peered":"2023-09-18T20:45:20.625971+0000","last_clean":"2023-09-18T20:45:20.625971+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:45:20.625971+0000","last_undegraded":"2023-09-18T20:45:20.625971+0000","last_fullsized":"2023-09-18T20:45:20.625971+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5928","last_scrub_stamp":"2023-09-18T20:45:20.625896+0000","last_deep_scrub":"32'5928","last_deep_scrub_stamp":"2023-09-18T20:45:20.625896+0000","last_clean_scrub_stamp":"2023-09-18T20:45:20.625896+0000","objects_scrubbed":36,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:39:44.176393+0000","scrub_duration":0.43503672700000001,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5848,"num_read_kb":23181,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":129,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5576","reported_seq":11080,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787886+0000","last_change":"2023-09-18T20:40:40.832137+0000","last_active":"2023-09-18T20:45:00.787886+0000","last_peered":"2023-09-18T20:45:00.787886+0000","last_clean":"2023-09-18T20:45:00.787886+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:45:00.787886+0000","last_undegraded":"2023-09-18T20:45:00.787886+0000","last_fullsized":"2023-09-18T20:45:00.787886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"32'4836","reported_seq":9931,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:21.049244+0000","last_change":"2023-09-18T20:45:21.049323+0000","last_active":"2023-09-18T20:45:21.049244+0000","last_peered":"2023-09-18T20:45:21.049244+0000","last_clean":"2023-09-18T20:45:21.049244+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:45:21.049244+0000","last_undegraded":"2023-09-18T20:45:21.049244+0000","last_fullsized":"2023-09-18T20:45:21.049244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4836","last_scrub_stamp":"2023-09-18T20:45:21.049147+0000","last_deep_scrub":"32'4836","last_deep_scrub_stamp":"2023-09-18T20:45:21.049147+0000","last_clean_scrub_stamp":"2023-09-18T20:45:21.049147+0000","objects_scrubbed":33,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:29:18.499117+0000","scrub_duration":0.37251760299999997,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124866560,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5102,"num_read_kb":20381,"num_write":4768,"num_write_kb":138225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":454,"num_omap_keys":13,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.203501+0000","last_change":"2023-09-18T20:45:00.203501+0000","last_active":"2023-09-18T20:45:00.203501+0000","last_peered":"2023-09-18T20:45:00.203501+0000","last_clean":"2023-09-18T20:45:00.203501+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:45:00.203501+0000","last_undegraded":"2023-09-18T20:45:00.203501+0000","last_fullsized":"2023-09-18T20:45:00.203501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:00.203367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:00.203367+0000","last_clean_scrub_stamp":"2023-09-18T20:45:00.203367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:09:19.993921+0000","scrub_duration":0.0012992380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:51.192940+0000","last_change":"2023-09-18T20:44:51.192940+0000","last_active":"2023-09-18T20:44:51.192940+0000","last_peered":"2023-09-18T20:44:51.192940+0000","last_clean":"2023-09-18T20:44:51.192940+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:44:51.192940+0000","last_undegraded":"2023-09-18T20:44:51.192940+0000","last_fullsized":"2023-09-18T20:44:51.192940+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:51.192786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:51.192786+0000","last_clean_scrub_stamp":"2023-09-18T20:44:51.192786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:12:10.502907+0000","scrub_duration":0.001553362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"32'4694","reported_seq":9409,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.809715+0000","last_change":"2023-09-18T20:44:53.516136+0000","last_active":"2023-09-18T20:45:00.809715+0000","last_peered":"2023-09-18T20:45:00.809715+0000","last_clean":"2023-09-18T20:45:00.809715+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:45:00.809715+0000","last_undegraded":"2023-09-18T20:45:00.809715+0000","last_fullsized":"2023-09-18T20:45:00.809715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4692","last_scrub_stamp":"2023-09-18T20:44:53.515992+0000","last_deep_scrub":"31'4692","last_deep_scrub_stamp":"2023-09-18T20:44:53.515992+0000","last_clean_scrub_stamp":"2023-09-18T20:44:53.515992+0000","objects_scrubbed":32,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:17:08.007446+0000","scrub_duration":0.339401179,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997668,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4616,"num_write_kb":133030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'7540","reported_seq":15118,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797340+0000","last_change":"2023-09-18T20:40:40.791026+0000","last_active":"2023-09-18T20:45:00.797340+0000","last_peered":"2023-09-18T20:45:00.797340+0000","last_clean":"2023-09-18T20:45:00.797340+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:45:00.797340+0000","last_undegraded":"2023-09-18T20:45:00.797340+0000","last_fullsized":"2023-09-18T20:45:00.797340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:08.192786+0000","last_change":"2023-09-18T20:45:08.192842+0000","last_active":"2023-09-18T20:45:08.192786+0000","last_peered":"2023-09-18T20:45:08.192786+0000","last_clean":"2023-09-18T20:45:08.192786+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:45:08.192786+0000","last_undegraded":"2023-09-18T20:45:08.192786+0000","last_fullsized":"2023-09-18T20:45:08.192786+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:08.192670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:08.192670+0000","last_clean_scrub_stamp":"2023-09-18T20:45:08.192670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:25:36.305554+0000","scrub_duration":0.0013098350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6544","reported_seq":12935,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787507+0000","last_change":"2023-09-18T20:44:56.214593+0000","last_active":"2023-09-18T20:45:00.787507+0000","last_peered":"2023-09-18T20:45:00.787507+0000","last_clean":"2023-09-18T20:45:00.787507+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:45:00.787507+0000","last_undegraded":"2023-09-18T20:45:00.787507+0000","last_fullsized":"2023-09-18T20:45:00.787507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6544","last_scrub_stamp":"2023-09-18T20:44:56.214397+0000","last_deep_scrub":"31'6544","last_deep_scrub_stamp":"2023-09-18T20:44:56.214397+0000","last_clean_scrub_stamp":"2023-09-18T20:44:56.214397+0000","objects_scrubbed":39,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:25:51.626829+0000","scrub_duration":0.4789736,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783348+0000","last_change":"2023-09-18T20:44:47.320025+0000","last_active":"2023-09-18T20:45:00.783348+0000","last_peered":"2023-09-18T20:45:00.783348+0000","last_clean":"2023-09-18T20:45:00.783348+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:45:00.783348+0000","last_undegraded":"2023-09-18T20:45:00.783348+0000","last_fullsized":"2023-09-18T20:45:00.783348+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:47.319876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:47.319876+0000","last_clean_scrub_stamp":"2023-09-18T20:44:47.319876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:33:13.500197+0000","scrub_duration":0.001796179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'4259","reported_seq":8458,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796900+0000","last_change":"2023-09-18T20:40:40.790444+0000","last_active":"2023-09-18T20:45:00.796900+0000","last_peered":"2023-09-18T20:45:00.796900+0000","last_clean":"2023-09-18T20:45:00.796900+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:45:00.796900+0000","last_undegraded":"2023-09-18T20:45:00.796900+0000","last_fullsized":"2023-09-18T20:45:00.796900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'99","reported_seq":172,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796819+0000","last_change":"2023-09-18T20:44:50.174363+0000","last_active":"2023-09-18T20:45:00.796819+0000","last_peered":"2023-09-18T20:45:00.796819+0000","last_clean":"2023-09-18T20:45:00.796819+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:45:00.796819+0000","last_undegraded":"2023-09-18T20:45:00.796819+0000","last_fullsized":"2023-09-18T20:45:00.796819+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'99","last_scrub_stamp":"2023-09-18T20:44:50.174236+0000","last_deep_scrub":"31'99","last_deep_scrub_stamp":"2023-09-18T20:44:50.174236+0000","last_clean_scrub_stamp":"2023-09-18T20:44:50.174236+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:53:49.471937+0000","scrub_duration":0.0039956039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:04.206942+0000","last_change":"2023-09-18T20:45:04.207025+0000","last_active":"2023-09-18T20:45:04.206942+0000","last_peered":"2023-09-18T20:45:04.206942+0000","last_clean":"2023-09-18T20:45:04.206942+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:45:04.206942+0000","last_undegraded":"2023-09-18T20:45:04.206942+0000","last_fullsized":"2023-09-18T20:45:04.206942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:04.206812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:04.206812+0000","last_clean_scrub_stamp":"2023-09-18T20:45:04.206812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:45:41.587932+0000","scrub_duration":0.0018374820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'5837","reported_seq":11566,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796748+0000","last_change":"2023-09-18T20:40:40.790900+0000","last_active":"2023-09-18T20:45:00.796748+0000","last_peered":"2023-09-18T20:45:00.796748+0000","last_clean":"2023-09-18T20:45:00.796748+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:45:00.796748+0000","last_undegraded":"2023-09-18T20:45:00.796748+0000","last_fullsized":"2023-09-18T20:45:00.796748+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:50.240541+0000","last_change":"2023-09-18T20:44:50.240541+0000","last_active":"2023-09-18T20:44:50.240541+0000","last_peered":"2023-09-18T20:44:50.240541+0000","last_clean":"2023-09-18T20:44:50.240541+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:44:50.240541+0000","last_undegraded":"2023-09-18T20:44:50.240541+0000","last_fullsized":"2023-09-18T20:44:50.240541+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:50.240423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:50.240423+0000","last_clean_scrub_stamp":"2023-09-18T20:44:50.240423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:40:37.117429+0000","scrub_duration":0.001250791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"31'5805","reported_seq":11506,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787794+0000","last_change":"2023-09-18T20:40:40.832355+0000","last_active":"2023-09-18T20:45:00.787794+0000","last_peered":"2023-09-18T20:45:00.787794+0000","last_clean":"2023-09-18T20:45:00.787794+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:45:00.787794+0000","last_undegraded":"2023-09-18T20:45:00.787794+0000","last_fullsized":"2023-09-18T20:45:00.787794+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:49.758943+0000","last_change":"2023-09-18T20:44:49.758943+0000","last_active":"2023-09-18T20:44:49.758943+0000","last_peered":"2023-09-18T20:44:49.758943+0000","last_clean":"2023-09-18T20:44:49.758943+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:44:49.758943+0000","last_undegraded":"2023-09-18T20:44:49.758943+0000","last_fullsized":"2023-09-18T20:44:49.758943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:49.758824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:49.758824+0000","last_clean_scrub_stamp":"2023-09-18T20:44:49.758824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:13:52.512458+0000","scrub_duration":0.0012109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6207","reported_seq":12324,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783860+0000","last_change":"2023-09-18T20:44:55.635468+0000","last_active":"2023-09-18T20:45:00.783860+0000","last_peered":"2023-09-18T20:45:00.783860+0000","last_clean":"2023-09-18T20:45:00.783860+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:45:00.783860+0000","last_undegraded":"2023-09-18T20:45:00.783860+0000","last_fullsized":"2023-09-18T20:45:00.783860+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6207","last_scrub_stamp":"2023-09-18T20:44:55.635254+0000","last_deep_scrub":"31'6207","last_deep_scrub_stamp":"2023-09-18T20:44:55.635254+0000","last_clean_scrub_stamp":"2023-09-18T20:44:55.635254+0000","objects_scrubbed":37,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:04:59.882463+0000","scrub_duration":0.45434064299999999,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:03.187558+0000","last_change":"2023-09-18T20:45:03.187604+0000","last_active":"2023-09-18T20:45:03.187558+0000","last_peered":"2023-09-18T20:45:03.187558+0000","last_clean":"2023-09-18T20:45:03.187558+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:45:03.187558+0000","last_undegraded":"2023-09-18T20:45:03.187558+0000","last_fullsized":"2023-09-18T20:45:03.187558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:03.187453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:03.187453+0000","last_clean_scrub_stamp":"2023-09-18T20:45:03.187453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:02:59.484397+0000","scrub_duration":0.0016553130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"32'5270","reported_seq":10465,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:09.542510+0000","last_change":"2023-09-18T20:45:09.542566+0000","last_active":"2023-09-18T20:45:09.542510+0000","last_peered":"2023-09-18T20:45:09.542510+0000","last_clean":"2023-09-18T20:45:09.542510+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:45:09.542510+0000","last_undegraded":"2023-09-18T20:45:09.542510+0000","last_fullsized":"2023-09-18T20:45:09.542510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5270","last_scrub_stamp":"2023-09-18T20:45:09.542437+0000","last_deep_scrub":"32'5270","last_deep_scrub_stamp":"2023-09-18T20:45:09.542437+0000","last_clean_scrub_stamp":"2023-09-18T20:45:09.542437+0000","objects_scrubbed":32,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:17:34.240394+0000","scrub_duration":0.37833424599999999,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"32'3845","reported_seq":7622,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:12.215345+0000","last_change":"2023-09-18T20:45:05.985307+0000","last_active":"2023-09-18T20:45:12.215345+0000","last_peered":"2023-09-18T20:45:12.215345+0000","last_clean":"2023-09-18T20:45:12.215345+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:45:12.215345+0000","last_undegraded":"2023-09-18T20:45:12.215345+0000","last_fullsized":"2023-09-18T20:45:12.215345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3845","last_scrub_stamp":"2023-09-18T20:45:05.985140+0000","last_deep_scrub":"32'3845","last_deep_scrub_stamp":"2023-09-18T20:45:05.985140+0000","last_clean_scrub_stamp":"2023-09-18T20:45:05.985140+0000","objects_scrubbed":22,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:28:59.141917+0000","scrub_duration":0.24710554500000001,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3902,"num_read_kb":14521,"num_write":3569,"num_write_kb":100159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"32'7193","reported_seq":14054,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:12.215278+0000","last_change":"2023-09-18T20:45:05.293657+0000","last_active":"2023-09-18T20:45:12.215278+0000","last_peered":"2023-09-18T20:45:12.215278+0000","last_clean":"2023-09-18T20:45:12.215278+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:45:12.215278+0000","last_undegraded":"2023-09-18T20:45:12.215278+0000","last_fullsized":"2023-09-18T20:45:12.215278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7193","last_scrub_stamp":"2023-09-18T20:45:05.293502+0000","last_deep_scrub":"32'7193","last_deep_scrub_stamp":"2023-09-18T20:45:05.293502+0000","last_clean_scrub_stamp":"2023-09-18T20:45:05.293502+0000","objects_scrubbed":44,"log_size":7193,"log_dups_size":0,"ondisk_log_size":7193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:47:04.282723+0000","scrub_duration":0.55523153300000005,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":7072,"num_read_kb":27249,"num_write":6867,"num_write_kb":191659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":507,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"32'3602","reported_seq":7133,"reported_epoch":32,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-18T20:45:22.425857+0000","last_change":"2023-09-18T20:45:22.173612+0000","last_active":"2023-09-18T20:45:22.425857+0000","last_peered":"2023-09-18T20:45:22.425857+0000","last_clean":"2023-09-18T20:45:22.425857+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:45:22.425857+0000","last_undegraded":"2023-09-18T20:45:22.425857+0000","last_fullsized":"2023-09-18T20:45:22.425857+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":21,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"32'4663","reported_seq":9100,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:19.007403+0000","last_change":"2023-09-18T20:45:19.007493+0000","last_active":"2023-09-18T20:45:19.007403+0000","last_peered":"2023-09-18T20:45:19.007403+0000","last_clean":"2023-09-18T20:45:19.007403+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:45:19.007403+0000","last_undegraded":"2023-09-18T20:45:19.007403+0000","last_fullsized":"2023-09-18T20:45:19.007403+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4663","last_scrub_stamp":"2023-09-18T20:45:19.007285+0000","last_deep_scrub":"32'4663","last_deep_scrub_stamp":"2023-09-18T20:45:19.007285+0000","last_clean_scrub_stamp":"2023-09-18T20:45:19.007285+0000","objects_scrubbed":28,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T20:52:31.949093+0000","scrub_duration":0.34925243299999997,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108093440,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4560,"num_read_kb":17596,"num_write":4373,"num_write_kb":120495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"32'4140","reported_seq":8377,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:11.573695+0000","last_change":"2023-09-18T20:45:11.573695+0000","last_active":"2023-09-18T20:45:11.573695+0000","last_peered":"2023-09-18T20:45:11.573695+0000","last_clean":"2023-09-18T20:45:11.573695+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:45:11.573695+0000","last_undegraded":"2023-09-18T20:45:11.573695+0000","last_fullsized":"2023-09-18T20:45:11.573695+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4140","last_scrub_stamp":"2023-09-18T20:45:11.573556+0000","last_deep_scrub":"32'4140","last_deep_scrub_stamp":"2023-09-18T20:45:11.573556+0000","last_clean_scrub_stamp":"2023-09-18T20:45:11.573556+0000","objects_scrubbed":26,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:21:44.192203+0000","scrub_duration":0.306416574,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"32'6121","reported_seq":12240,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:14.708033+0000","last_change":"2023-09-18T20:45:14.708099+0000","last_active":"2023-09-18T20:45:14.708033+0000","last_peered":"2023-09-18T20:45:14.708033+0000","last_clean":"2023-09-18T20:45:14.708033+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:45:14.708033+0000","last_undegraded":"2023-09-18T20:45:14.708033+0000","last_fullsized":"2023-09-18T20:45:14.708033+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6121","last_scrub_stamp":"2023-09-18T20:45:14.707953+0000","last_deep_scrub":"32'6121","last_deep_scrub_stamp":"2023-09-18T20:45:14.707953+0000","last_clean_scrub_stamp":"2023-09-18T20:45:14.707953+0000","objects_scrubbed":37,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:28:20.542024+0000","scrub_duration":0.45992617699999999,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"32'4163","reported_seq":8206,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:12.215334+0000","last_change":"2023-09-18T20:40:40.833389+0000","last_active":"2023-09-18T20:45:12.215334+0000","last_peered":"2023-09-18T20:45:12.215334+0000","last_clean":"2023-09-18T20:45:12.215334+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:45:12.215334+0000","last_undegraded":"2023-09-18T20:45:12.215334+0000","last_fullsized":"2023-09-18T20:45:12.215334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4182,"num_read_kb":15599,"num_write":3896,"num_write_kb":109676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"31'4527","reported_seq":9046,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796709+0000","last_change":"2023-09-18T20:40:40.789756+0000","last_active":"2023-09-18T20:45:00.796709+0000","last_peered":"2023-09-18T20:45:00.796709+0000","last_clean":"2023-09-18T20:45:00.796709+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:45:00.796709+0000","last_undegraded":"2023-09-18T20:45:00.796709+0000","last_fullsized":"2023-09-18T20:45:00.796709+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"32'6271","reported_seq":12355,"reported_epoch":32,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-18T20:45:21.490728+0000","last_change":"2023-09-18T20:45:21.206905+0000","last_active":"2023-09-18T20:45:21.490728+0000","last_peered":"2023-09-18T20:45:21.490728+0000","last_clean":"2023-09-18T20:45:21.490728+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:45:21.490728+0000","last_undegraded":"2023-09-18T20:45:21.490728+0000","last_fullsized":"2023-09-18T20:45:21.490728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":24,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"32'5004","reported_seq":9981,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:13.065201+0000","last_change":"2023-09-18T20:45:13.065201+0000","last_active":"2023-09-18T20:45:13.065201+0000","last_peered":"2023-09-18T20:45:13.065201+0000","last_clean":"2023-09-18T20:45:13.065201+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:45:13.065201+0000","last_undegraded":"2023-09-18T20:45:13.065201+0000","last_fullsized":"2023-09-18T20:45:13.065201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5004","last_scrub_stamp":"2023-09-18T20:45:13.065001+0000","last_deep_scrub":"32'5004","last_deep_scrub_stamp":"2023-09-18T20:45:13.065001+0000","last_clean_scrub_stamp":"2023-09-18T20:45:13.065001+0000","objects_scrubbed":33,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:18:42.417094+0000","scrub_duration":0.371655493,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643108,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4927,"num_read_kb":19684,"num_write":4935,"num_write_kb":138349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":340,"num_omap_keys":5,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4608","reported_seq":9080,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797062+0000","last_change":"2023-09-18T20:44:49.462443+0000","last_active":"2023-09-18T20:45:00.797062+0000","last_peered":"2023-09-18T20:45:00.797062+0000","last_clean":"2023-09-18T20:45:00.797062+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:45:00.797062+0000","last_undegraded":"2023-09-18T20:45:00.797062+0000","last_fullsized":"2023-09-18T20:45:00.797062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4608","last_scrub_stamp":"2023-09-18T20:44:49.462306+0000","last_deep_scrub":"31'4608","last_deep_scrub_stamp":"2023-09-18T20:44:49.462306+0000","last_clean_scrub_stamp":"2023-09-18T20:44:49.462306+0000","objects_scrubbed":27,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:54:51.936441+0000","scrub_duration":0.337859574,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"32'4458","reported_seq":8876,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:18.595236+0000","last_change":"2023-09-18T20:45:18.595236+0000","last_active":"2023-09-18T20:45:18.595236+0000","last_peered":"2023-09-18T20:45:18.595236+0000","last_clean":"2023-09-18T20:45:18.595236+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:45:18.595236+0000","last_undegraded":"2023-09-18T20:45:18.595236+0000","last_fullsized":"2023-09-18T20:45:18.595236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4458","last_scrub_stamp":"2023-09-18T20:45:18.595102+0000","last_deep_scrub":"32'4458","last_deep_scrub_stamp":"2023-09-18T20:45:18.595102+0000","last_clean_scrub_stamp":"2023-09-18T20:45:18.595102+0000","objects_scrubbed":27,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:43:59.143459+0000","scrub_duration":0.39233341900000002,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"32'5131","reported_seq":10375,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:22.060175+0000","last_change":"2023-09-18T20:45:22.060238+0000","last_active":"2023-09-18T20:45:22.060175+0000","last_peered":"2023-09-18T20:45:22.060175+0000","last_clean":"2023-09-18T20:45:22.060175+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:45:22.060175+0000","last_undegraded":"2023-09-18T20:45:22.060175+0000","last_fullsized":"2023-09-18T20:45:22.060175+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5131","last_scrub_stamp":"2023-09-18T20:45:22.060098+0000","last_deep_scrub":"32'5131","last_deep_scrub_stamp":"2023-09-18T20:45:22.060098+0000","last_clean_scrub_stamp":"2023-09-18T20:45:22.060098+0000","objects_scrubbed":32,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:04:56.690721+0000","scrub_duration":0.37624659799999999,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"32'6125","reported_seq":11984,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:18.801680+0000","last_change":"2023-09-18T20:45:18.801745+0000","last_active":"2023-09-18T20:45:18.801680+0000","last_peered":"2023-09-18T20:45:18.801680+0000","last_clean":"2023-09-18T20:45:18.801680+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:45:18.801680+0000","last_undegraded":"2023-09-18T20:45:18.801680+0000","last_fullsized":"2023-09-18T20:45:18.801680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6125","last_scrub_stamp":"2023-09-18T20:45:18.801592+0000","last_deep_scrub":"32'6125","last_deep_scrub_stamp":"2023-09-18T20:45:18.801592+0000","last_clean_scrub_stamp":"2023-09-18T20:45:18.801592+0000","objects_scrubbed":36,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:41:37.387977+0000","scrub_duration":0.56301445699999997,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"32'6931","reported_seq":13698,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:12.215358+0000","last_change":"2023-09-18T20:40:40.833412+0000","last_active":"2023-09-18T20:45:12.215358+0000","last_peered":"2023-09-18T20:45:12.215358+0000","last_clean":"2023-09-18T20:45:12.215358+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:45:12.215358+0000","last_undegraded":"2023-09-18T20:45:12.215358+0000","last_fullsized":"2023-09-18T20:45:12.215358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6911,"num_read_kb":26471,"num_write":6659,"num_write_kb":186156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"32'6714","reported_seq":13453,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783375+0000","last_change":"2023-09-18T20:44:52.726361+0000","last_active":"2023-09-18T20:45:00.783375+0000","last_peered":"2023-09-18T20:45:00.783375+0000","last_clean":"2023-09-18T20:45:00.783375+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:45:00.783375+0000","last_undegraded":"2023-09-18T20:45:00.783375+0000","last_fullsized":"2023-09-18T20:45:00.783375+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6713","last_scrub_stamp":"2023-09-18T20:44:52.726150+0000","last_deep_scrub":"31'6713","last_deep_scrub_stamp":"2023-09-18T20:44:52.726150+0000","last_clean_scrub_stamp":"2023-09-18T20:44:52.726150+0000","objects_scrubbed":41,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:53:09.298051+0000","scrub_duration":0.52103732999999997,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390706,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6623,"num_write_kb":190145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'5371","reported_seq":10460,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796939+0000","last_change":"2023-09-18T20:40:40.789720+0000","last_active":"2023-09-18T20:45:00.796939+0000","last_peered":"2023-09-18T20:45:00.796939+0000","last_clean":"2023-09-18T20:45:00.796939+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:45:00.796939+0000","last_undegraded":"2023-09-18T20:45:00.796939+0000","last_fullsized":"2023-09-18T20:45:00.796939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"31'4666","reported_seq":9376,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.821187+0000","last_change":"2023-09-18T20:40:40.790245+0000","last_active":"2023-09-18T20:45:00.821187+0000","last_peered":"2023-09-18T20:45:00.821187+0000","last_clean":"2023-09-18T20:45:00.821187+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:45:00.821187+0000","last_undegraded":"2023-09-18T20:45:00.821187+0000","last_fullsized":"2023-09-18T20:45:00.821187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:40:37.764654+0000","last_clean_scrub_stamp":"2023-09-18T20:40:37.764654+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4670,"num_read_kb":18910,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216328790,"num_objects":1059,"num_object_clones":0,"num_object_copies":2118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":169470,"num_read_kb":673117,"num_write":169074,"num_write_kb":4770857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2446,"num_omap_keys":58,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392695808,"data_stored":8400299180,"data_compressed":322494,"data_compressed_allocated":2187264,"data_compressed_original":9920512,"omap_allocated":0,"internal_metadata":0},"log_size":172243,"ondisk_log_size":172243,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2816120,"kb_used_data":2397808,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91555720,"statfs":{"total":96636764160,"available":93753057280,"internally_reserved":0,"allocated":2455355392,"data_stored":2456650470,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0660000000000001,"15min":0.55100000000000005},"min":{"1min":0.16700000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":3.4380000000000002,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.04,"15min":0.54900000000000004},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.5730000000000004,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Sep 18 20:44:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.3260000000000001,"15min":0.64400000000000002},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.629,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.3160000000000001,"15min":0.65200000000000002},"min":{"1min":0.27100000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.6289999999999996,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.497}]}]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2978380,"kb_used_data":2532292,"kb_used_omap":1,"kb_used_meta":446078,"kb_avail":91393460,"statfs":{"total":96636764160,"available":93586903040,"internally_reserved":0,"allocated":2593067008,"data_stored":2595783988,"data_compressed":59035,"data_compressed_allocated":430080,"data_compressed_original":3358720,"omap_allocated":1594,"internal_metadata":456784326},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.46300000000000002,"15min":0.34699999999999998},"min":{"1min":0.18099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.59399999999999997,"15min":0.40500000000000003},"min":{"1min":0.189,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.0049999999999999,"5min":3.585,"15min":3.585},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Sep 18 20:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.39200000000000002},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5489999999999999,"5min":12.382,"15min":12.382},"last":0.315},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.53400000000000003,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3829492,"kb_used_data":3266796,"kb_used_omap":1,"kb_used_meta":562686,"kb_avail":90542348,"statfs":{"total":96636764160,"available":92715364352,"internally_reserved":0,"allocated":3345199104,"data_stored":3349353831,"data_compressed":165270,"data_compressed_allocated":1118208,"data_compressed_original":5496832,"omap_allocated":1594,"internal_metadata":576190918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80900000000000005,"15min":0.45700000000000002},"min":{"1min":0.23799999999999999,"5min":0.158,"15min":0.158},"max":{"1min":5.6150000000000002,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.9079999999999999,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Sep 18 20:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.94199999999999995,"15min":0.503},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.4710000000000001,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.1379999999999999,"15min":0.58399999999999996},"min":{"1min":0.26300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":5.5049999999999999,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592739328,"data_stored":2595090829,"data_compressed":53273,"data_compressed_allocated":393216,"data_compressed_original":2785280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344871424,"data_stored":3348660672,"data_compressed":159508,"data_compressed_allocated":1081344,"data_compressed_original":4923392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455085056,"data_stored":2456547679,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:45:26.147 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.147 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.148 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.148 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.149 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.149 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.150 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.150 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.151 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.151 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.152 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.152 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.152 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-09-18T20:40:37.764654+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=40, tm_sec=37, tm_wday=0, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=18, tm_hour=20, tm_min=44, tm_sec=42, tm_wday=0, tm_yday=261, tm_isdst=0) 2023-09-18T20:45:26.153 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-18T20:45:46.154 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-18T20:45:46.486 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-09-18T20:45:46.487 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2023-09-18T20:45:46.499 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2023-09-18T20:45:45.738266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4216935542,"num_objects":1063,"num_object_clones":2,"num_object_copies":2126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":169824,"num_read_kb":673418,"num_write":169214,"num_write_kb":4772767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3462,"num_omap_keys":78,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172730,"ondisk_log_size":172730,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9632336,"kb_used_data":8196920,"kb_used_omap":4,"kb_used_meta":1435387,"kb_avail":273483184,"statfs":{"total":289910292480,"available":280046780416,"internally_reserved":0,"allocated":8393646080,"data_stored":8404041089,"data_compressed":393110,"data_compressed_allocated":2269184,"data_compressed_original":12959744,"omap_allocated":4782,"internal_metadata":1469836626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":248,"num_read_kb":210,"num_write":3,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"3.1f","version":"32'5500","reported_seq":10850,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:25.025762+0000","last_change":"2023-09-18T20:45:25.025844+0000","last_active":"2023-09-18T20:45:25.025762+0000","last_peered":"2023-09-18T20:45:25.025762+0000","last_clean":"2023-09-18T20:45:25.025762+0000","last_became_active":"2023-09-18T20:40:38.789742+0000","last_became_peered":"2023-09-18T20:40:38.789742+0000","last_unstale":"2023-09-18T20:45:25.025762+0000","last_undegraded":"2023-09-18T20:45:25.025762+0000","last_fullsized":"2023-09-18T20:45:25.025762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5500","last_scrub_stamp":"2023-09-18T20:45:25.025661+0000","last_deep_scrub":"32'5500","last_deep_scrub_stamp":"2023-09-18T20:45:25.025661+0000","last_clean_scrub_stamp":"2023-09-18T20:45:25.025661+0000","objects_scrubbed":33,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:39:25.057401+0000","scrub_duration":0.42665703300000002,"objects_trimmed":0,"snaptrim_duration":0.00055673599999999995,"stat_sum":{"num_bytes":134217778,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5300,"num_read_kb":21200,"num_write":5429,"num_write_kb":152609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"32'4999","reported_seq":9822,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:05.570688+0000","last_change":"2023-09-18T20:45:05.570748+0000","last_active":"2023-09-18T20:45:05.570688+0000","last_peered":"2023-09-18T20:45:05.570688+0000","last_clean":"2023-09-18T20:45:05.570688+0000","last_became_active":"2023-09-18T20:40:38.786899+0000","last_became_peered":"2023-09-18T20:40:38.786899+0000","last_unstale":"2023-09-18T20:45:05.570688+0000","last_undegraded":"2023-09-18T20:45:05.570688+0000","last_fullsized":"2023-09-18T20:45:05.570688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4999","last_scrub_stamp":"2023-09-18T20:45:05.570612+0000","last_deep_scrub":"32'4999","last_deep_scrub_stamp":"2023-09-18T20:45:05.570612+0000","last_clean_scrub_stamp":"2023-09-18T20:45:05.570612+0000","objects_scrubbed":30,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:08:24.598326+0000","scrub_duration":0.41260589600000003,"objects_trimmed":0,"snaptrim_duration":0.00094968500000000003,"stat_sum":{"num_bytes":122253312,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4773,"num_read_kb":19092,"num_write":4927,"num_write_kb":138348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"32'5748","reported_seq":11466,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:07.609758+0000","last_change":"2023-09-18T20:45:07.609758+0000","last_active":"2023-09-18T20:45:07.609758+0000","last_peered":"2023-09-18T20:45:07.609758+0000","last_clean":"2023-09-18T20:45:07.609758+0000","last_became_active":"2023-09-18T20:40:38.785420+0000","last_became_peered":"2023-09-18T20:40:38.785420+0000","last_unstale":"2023-09-18T20:45:07.609758+0000","last_undegraded":"2023-09-18T20:45:07.609758+0000","last_fullsized":"2023-09-18T20:45:07.609758+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5748","last_scrub_stamp":"2023-09-18T20:45:07.609608+0000","last_deep_scrub":"32'5748","last_deep_scrub_stamp":"2023-09-18T20:45:07.609608+0000","last_clean_scrub_stamp":"2023-09-18T20:45:07.609608+0000","objects_scrubbed":36,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:43:28.999233+0000","scrub_duration":0.43395783999999998,"objects_trimmed":0,"snaptrim_duration":0.00073765199999999999,"stat_sum":{"num_bytes":142614578,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5668,"num_read_kb":22672,"num_write":5660,"num_write_kb":161725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"32'5928","reported_seq":11822,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.262026+0000","last_change":"2023-09-18T20:45:20.626030+0000","last_active":"2023-09-18T20:45:33.262026+0000","last_peered":"2023-09-18T20:45:33.262026+0000","last_clean":"2023-09-18T20:45:33.262026+0000","last_became_active":"2023-09-18T20:40:38.787801+0000","last_became_peered":"2023-09-18T20:40:38.787801+0000","last_unstale":"2023-09-18T20:45:33.262026+0000","last_undegraded":"2023-09-18T20:45:33.262026+0000","last_fullsized":"2023-09-18T20:45:33.262026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5928","last_scrub_stamp":"2023-09-18T20:45:20.625896+0000","last_deep_scrub":"32'5928","last_deep_scrub_stamp":"2023-09-18T20:45:20.625896+0000","last_clean_scrub_stamp":"2023-09-18T20:45:20.625896+0000","objects_scrubbed":36,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:39:44.176393+0000","scrub_duration":0.43503672700000001,"objects_trimmed":0,"snaptrim_duration":0.001242981,"stat_sum":{"num_bytes":142606352,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5855,"num_read_kb":23187,"num_write":5855,"num_write_kb":162490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":129,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"32'5611","reported_seq":11125,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:32.048844+0000","last_change":"2023-09-18T20:45:32.048844+0000","last_active":"2023-09-18T20:45:32.048844+0000","last_peered":"2023-09-18T20:45:32.048844+0000","last_clean":"2023-09-18T20:45:32.048844+0000","last_became_active":"2023-09-18T20:40:38.791077+0000","last_became_peered":"2023-09-18T20:40:38.791077+0000","last_unstale":"2023-09-18T20:45:32.048844+0000","last_undegraded":"2023-09-18T20:45:32.048844+0000","last_fullsized":"2023-09-18T20:45:32.048844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5611","last_scrub_stamp":"2023-09-18T20:45:32.048706+0000","last_deep_scrub":"32'5611","last_deep_scrub_stamp":"2023-09-18T20:45:32.048706+0000","last_clean_scrub_stamp":"2023-09-18T20:45:32.048706+0000","objects_scrubbed":35,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:03:59.544936+0000","scrub_duration":0.42199666600000002,"objects_trimmed":0,"snaptrim_duration":0.00051718300000000005,"stat_sum":{"num_bytes":141111296,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5465,"num_read_kb":22297,"num_write":5526,"num_write_kb":157644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"32'4838","reported_seq":9944,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.261063+0000","last_change":"2023-09-18T20:45:21.049323+0000","last_active":"2023-09-18T20:45:33.261063+0000","last_peered":"2023-09-18T20:45:33.261063+0000","last_clean":"2023-09-18T20:45:33.261063+0000","last_became_active":"2023-09-18T20:40:38.784356+0000","last_became_peered":"2023-09-18T20:40:38.784356+0000","last_unstale":"2023-09-18T20:45:33.261063+0000","last_undegraded":"2023-09-18T20:45:33.261063+0000","last_fullsized":"2023-09-18T20:45:33.261063+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4836","last_scrub_stamp":"2023-09-18T20:45:21.049147+0000","last_deep_scrub":"32'4836","last_deep_scrub_stamp":"2023-09-18T20:45:21.049147+0000","last_clean_scrub_stamp":"2023-09-18T20:45:21.049147+0000","objects_scrubbed":33,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:29:18.499117+0000","scrub_duration":0.37251760299999997,"objects_trimmed":0,"snaptrim_duration":0.000535558,"stat_sum":{"num_bytes":124870656,"num_objects":34,"num_object_clones":1,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5126,"num_read_kb":20403,"num_write":4769,"num_write_kb":138229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":454,"num_omap_keys":13,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.203501+0000","last_change":"2023-09-18T20:45:00.203501+0000","last_active":"2023-09-18T20:45:00.203501+0000","last_peered":"2023-09-18T20:45:00.203501+0000","last_clean":"2023-09-18T20:45:00.203501+0000","last_became_active":"2023-09-18T20:39:18.754293+0000","last_became_peered":"2023-09-18T20:39:18.754293+0000","last_unstale":"2023-09-18T20:45:00.203501+0000","last_undegraded":"2023-09-18T20:45:00.203501+0000","last_fullsized":"2023-09-18T20:45:00.203501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:00.203367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:00.203367+0000","last_clean_scrub_stamp":"2023-09-18T20:45:00.203367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:09:19.993921+0000","scrub_duration":0.0012992380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-09-18T20:44:51.192940+0000","last_change":"2023-09-18T20:44:51.192940+0000","last_active":"2023-09-18T20:44:51.192940+0000","last_peered":"2023-09-18T20:44:51.192940+0000","last_clean":"2023-09-18T20:44:51.192940+0000","last_became_active":"2023-09-18T20:39:18.754304+0000","last_became_peered":"2023-09-18T20:39:18.754304+0000","last_unstale":"2023-09-18T20:44:51.192940+0000","last_undegraded":"2023-09-18T20:44:51.192940+0000","last_fullsized":"2023-09-18T20:44:51.192940+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:51.192786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:51.192786+0000","last_clean_scrub_stamp":"2023-09-18T20:44:51.192786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:12:10.502907+0000","scrub_duration":0.001553362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"32'4694","reported_seq":9409,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.809715+0000","last_change":"2023-09-18T20:44:53.516136+0000","last_active":"2023-09-18T20:45:00.809715+0000","last_peered":"2023-09-18T20:45:00.809715+0000","last_clean":"2023-09-18T20:45:00.809715+0000","last_became_active":"2023-09-18T20:40:38.784218+0000","last_became_peered":"2023-09-18T20:40:38.784218+0000","last_unstale":"2023-09-18T20:45:00.809715+0000","last_undegraded":"2023-09-18T20:45:00.809715+0000","last_fullsized":"2023-09-18T20:45:00.809715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4692","last_scrub_stamp":"2023-09-18T20:44:53.515992+0000","last_deep_scrub":"31'4692","last_deep_scrub_stamp":"2023-09-18T20:44:53.515992+0000","last_clean_scrub_stamp":"2023-09-18T20:44:53.515992+0000","objects_scrubbed":32,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:17:08.007446+0000","scrub_duration":0.339401179,"objects_trimmed":0,"snaptrim_duration":0.0012270720000000001,"stat_sum":{"num_bytes":117997668,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":4665,"num_read_kb":18660,"num_write":4616,"num_write_kb":133030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"32'7586","reported_seq":15173,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:24.823899+0000","last_change":"2023-09-18T20:45:24.824019+0000","last_active":"2023-09-18T20:45:24.823899+0000","last_peered":"2023-09-18T20:45:24.823899+0000","last_clean":"2023-09-18T20:45:24.823899+0000","last_became_active":"2023-09-18T20:40:38.784117+0000","last_became_peered":"2023-09-18T20:40:38.784117+0000","last_unstale":"2023-09-18T20:45:24.823899+0000","last_undegraded":"2023-09-18T20:45:24.823899+0000","last_fullsized":"2023-09-18T20:45:24.823899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7586","last_scrub_stamp":"2023-09-18T20:45:24.823776+0000","last_deep_scrub":"32'7586","last_deep_scrub_stamp":"2023-09-18T20:45:24.823776+0000","last_clean_scrub_stamp":"2023-09-18T20:45:24.823776+0000","objects_scrubbed":46,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:05:34.192365+0000","scrub_duration":0.62617858199999998,"objects_trimmed":0,"snaptrim_duration":0.0013732250000000001,"stat_sum":{"num_bytes":185102352,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7543,"num_read_kb":30124,"num_write":7492,"num_write_kb":210213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:08.192842+0000","last_change":"2023-09-18T20:45:08.192842+0000","last_active":"2023-09-18T20:45:08.192842+0000","last_peered":"2023-09-18T20:45:08.192842+0000","last_clean":"2023-09-18T20:45:08.192842+0000","last_became_active":"2023-09-18T20:39:18.754022+0000","last_became_peered":"2023-09-18T20:39:18.754022+0000","last_unstale":"2023-09-18T20:45:08.192842+0000","last_undegraded":"2023-09-18T20:45:08.192842+0000","last_fullsized":"2023-09-18T20:45:08.192842+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:08.192670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:08.192670+0000","last_clean_scrub_stamp":"2023-09-18T20:45:08.192670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:25:36.305554+0000","scrub_duration":0.0013098350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'6544","reported_seq":12935,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787507+0000","last_change":"2023-09-18T20:44:56.214593+0000","last_active":"2023-09-18T20:45:00.787507+0000","last_peered":"2023-09-18T20:45:00.787507+0000","last_clean":"2023-09-18T20:45:00.787507+0000","last_became_active":"2023-09-18T20:40:38.782694+0000","last_became_peered":"2023-09-18T20:40:38.782694+0000","last_unstale":"2023-09-18T20:45:00.787507+0000","last_undegraded":"2023-09-18T20:45:00.787507+0000","last_fullsized":"2023-09-18T20:45:00.787507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6544","last_scrub_stamp":"2023-09-18T20:44:56.214397+0000","last_deep_scrub":"31'6544","last_deep_scrub_stamp":"2023-09-18T20:44:56.214397+0000","last_clean_scrub_stamp":"2023-09-18T20:44:56.214397+0000","objects_scrubbed":39,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:25:51.626829+0000","scrub_duration":0.4789736,"objects_trimmed":0,"snaptrim_duration":0.00038837499999999999,"stat_sum":{"num_bytes":159916032,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6341,"num_read_kb":25364,"num_write":6463,"num_write_kb":181316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783348+0000","last_change":"2023-09-18T20:44:47.320025+0000","last_active":"2023-09-18T20:45:00.783348+0000","last_peered":"2023-09-18T20:45:00.783348+0000","last_clean":"2023-09-18T20:45:00.783348+0000","last_became_active":"2023-09-18T20:39:18.753214+0000","last_became_peered":"2023-09-18T20:39:18.753214+0000","last_unstale":"2023-09-18T20:45:00.783348+0000","last_undegraded":"2023-09-18T20:45:00.783348+0000","last_fullsized":"2023-09-18T20:45:00.783348+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:47.319876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:47.319876+0000","last_clean_scrub_stamp":"2023-09-18T20:44:47.319876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:33:13.500197+0000","scrub_duration":0.001796179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"32'4284","reported_seq":8493,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:32.537909+0000","last_change":"2023-09-18T20:45:32.537962+0000","last_active":"2023-09-18T20:45:32.537909+0000","last_peered":"2023-09-18T20:45:32.537909+0000","last_clean":"2023-09-18T20:45:32.537909+0000","last_became_active":"2023-09-18T20:40:38.784922+0000","last_became_peered":"2023-09-18T20:40:38.784922+0000","last_unstale":"2023-09-18T20:45:32.537909+0000","last_undegraded":"2023-09-18T20:45:32.537909+0000","last_fullsized":"2023-09-18T20:45:32.537909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4284","last_scrub_stamp":"2023-09-18T20:45:32.537829+0000","last_deep_scrub":"32'4284","last_deep_scrub_stamp":"2023-09-18T20:45:32.537829+0000","last_clean_scrub_stamp":"2023-09-18T20:45:32.537829+0000","objects_scrubbed":25,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:02:39.757100+0000","scrub_duration":0.34073997900000003,"objects_trimmed":0,"snaptrim_duration":0.001295547,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4159,"num_read_kb":16636,"num_write":4227,"num_write_kb":119176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'99","reported_seq":172,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.796819+0000","last_change":"2023-09-18T20:44:50.174363+0000","last_active":"2023-09-18T20:45:00.796819+0000","last_peered":"2023-09-18T20:45:00.796819+0000","last_clean":"2023-09-18T20:45:00.796819+0000","last_became_active":"2023-09-18T20:39:12.867378+0000","last_became_peered":"2023-09-18T20:39:12.867378+0000","last_unstale":"2023-09-18T20:45:00.796819+0000","last_undegraded":"2023-09-18T20:45:00.796819+0000","last_fullsized":"2023-09-18T20:45:00.796819+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'99","last_scrub_stamp":"2023-09-18T20:44:50.174236+0000","last_deep_scrub":"31'99","last_deep_scrub_stamp":"2023-09-18T20:44:50.174236+0000","last_clean_scrub_stamp":"2023-09-18T20:44:50.174236+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:53:49.471937+0000","scrub_duration":0.0039956039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:04.207025+0000","last_change":"2023-09-18T20:45:04.207025+0000","last_active":"2023-09-18T20:45:04.207025+0000","last_peered":"2023-09-18T20:45:04.207025+0000","last_clean":"2023-09-18T20:45:04.207025+0000","last_became_active":"2023-09-18T20:39:18.753043+0000","last_became_peered":"2023-09-18T20:39:18.753043+0000","last_unstale":"2023-09-18T20:45:04.207025+0000","last_undegraded":"2023-09-18T20:45:04.207025+0000","last_fullsized":"2023-09-18T20:45:04.207025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:04.206812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:04.206812+0000","last_clean_scrub_stamp":"2023-09-18T20:45:04.206812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T07:45:41.587932+0000","scrub_duration":0.0018374820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"32'5873","reported_seq":11611,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:26.631640+0000","last_change":"2023-09-18T20:45:26.631730+0000","last_active":"2023-09-18T20:45:26.631640+0000","last_peered":"2023-09-18T20:45:26.631640+0000","last_clean":"2023-09-18T20:45:26.631640+0000","last_became_active":"2023-09-18T20:40:38.783787+0000","last_became_peered":"2023-09-18T20:40:38.783787+0000","last_unstale":"2023-09-18T20:45:26.631640+0000","last_undegraded":"2023-09-18T20:45:26.631640+0000","last_fullsized":"2023-09-18T20:45:26.631640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5873","last_scrub_stamp":"2023-09-18T20:45:26.631526+0000","last_deep_scrub":"32'5873","last_deep_scrub_stamp":"2023-09-18T20:45:26.631526+0000","last_clean_scrub_stamp":"2023-09-18T20:45:26.631526+0000","objects_scrubbed":36,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:07:17.252271+0000","scrub_duration":0.41981487200000001,"objects_trimmed":0,"snaptrim_duration":0.001436765,"stat_sum":{"num_bytes":143126528,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5689,"num_read_kb":22756,"num_write":5790,"num_write_kb":162444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783633+0000","last_change":"2023-09-18T20:44:50.240541+0000","last_active":"2023-09-18T20:45:00.783633+0000","last_peered":"2023-09-18T20:45:00.783633+0000","last_clean":"2023-09-18T20:45:00.783633+0000","last_became_active":"2023-09-18T20:39:18.753178+0000","last_became_peered":"2023-09-18T20:39:18.753178+0000","last_unstale":"2023-09-18T20:45:00.783633+0000","last_undegraded":"2023-09-18T20:45:00.783633+0000","last_fullsized":"2023-09-18T20:45:00.783633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:50.240423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:50.240423+0000","last_clean_scrub_stamp":"2023-09-18T20:44:50.240423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:40:37.117429+0000","scrub_duration":0.001250791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"32'5839","reported_seq":11549,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:30.005086+0000","last_change":"2023-09-18T20:45:30.005152+0000","last_active":"2023-09-18T20:45:30.005086+0000","last_peered":"2023-09-18T20:45:30.005086+0000","last_clean":"2023-09-18T20:45:30.005086+0000","last_became_active":"2023-09-18T20:40:38.786110+0000","last_became_peered":"2023-09-18T20:40:38.786110+0000","last_unstale":"2023-09-18T20:45:30.005086+0000","last_undegraded":"2023-09-18T20:45:30.005086+0000","last_fullsized":"2023-09-18T20:45:30.005086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5839","last_scrub_stamp":"2023-09-18T20:45:30.005004+0000","last_deep_scrub":"32'5839","last_deep_scrub_stamp":"2023-09-18T20:45:30.005004+0000","last_clean_scrub_stamp":"2023-09-18T20:45:30.005004+0000","objects_scrubbed":34,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:24:03.355946+0000","scrub_duration":0.42372552299999999,"objects_trimmed":0,"snaptrim_duration":0.000727475,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":5661,"num_read_kb":22644,"num_write":5758,"num_write_kb":162124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.787346+0000","last_change":"2023-09-18T20:44:49.758943+0000","last_active":"2023-09-18T20:45:00.787346+0000","last_peered":"2023-09-18T20:45:00.787346+0000","last_clean":"2023-09-18T20:45:00.787346+0000","last_became_active":"2023-09-18T20:39:18.753077+0000","last_became_peered":"2023-09-18T20:39:18.753077+0000","last_unstale":"2023-09-18T20:45:00.787346+0000","last_undegraded":"2023-09-18T20:45:00.787346+0000","last_fullsized":"2023-09-18T20:45:00.787346+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:44:49.758824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:44:49.758824+0000","last_clean_scrub_stamp":"2023-09-18T20:44:49.758824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:13:52.512458+0000","scrub_duration":0.0012109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'6207","reported_seq":12324,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.783860+0000","last_change":"2023-09-18T20:44:55.635468+0000","last_active":"2023-09-18T20:45:00.783860+0000","last_peered":"2023-09-18T20:45:00.783860+0000","last_clean":"2023-09-18T20:45:00.783860+0000","last_became_active":"2023-09-18T20:40:38.783790+0000","last_became_peered":"2023-09-18T20:40:38.783790+0000","last_unstale":"2023-09-18T20:45:00.783860+0000","last_undegraded":"2023-09-18T20:45:00.783860+0000","last_fullsized":"2023-09-18T20:45:00.783860+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6207","last_scrub_stamp":"2023-09-18T20:44:55.635254+0000","last_deep_scrub":"31'6207","last_deep_scrub_stamp":"2023-09-18T20:44:55.635254+0000","last_clean_scrub_stamp":"2023-09-18T20:44:55.635254+0000","objects_scrubbed":37,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T22:04:59.882463+0000","scrub_duration":0.45434064299999999,"objects_trimmed":0,"snaptrim_duration":0.0010589390000000001,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6067,"num_read_kb":24268,"num_write":6121,"num_write_kb":171749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:03.187604+0000","last_change":"2023-09-18T20:45:03.187604+0000","last_active":"2023-09-18T20:45:03.187604+0000","last_peered":"2023-09-18T20:45:03.187604+0000","last_clean":"2023-09-18T20:45:03.187604+0000","last_became_active":"2023-09-18T20:39:18.754062+0000","last_became_peered":"2023-09-18T20:39:18.754062+0000","last_unstale":"2023-09-18T20:45:03.187604+0000","last_undegraded":"2023-09-18T20:45:03.187604+0000","last_fullsized":"2023-09-18T20:45:03.187604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-18T20:45:03.187453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-18T20:45:03.187453+0000","last_clean_scrub_stamp":"2023-09-18T20:45:03.187453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:02:59.484397+0000","scrub_duration":0.0016553130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"32'5270","reported_seq":10465,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:09.542510+0000","last_change":"2023-09-18T20:45:09.542566+0000","last_active":"2023-09-18T20:45:09.542510+0000","last_peered":"2023-09-18T20:45:09.542510+0000","last_clean":"2023-09-18T20:45:09.542510+0000","last_became_active":"2023-09-18T20:40:38.784959+0000","last_became_peered":"2023-09-18T20:40:38.784959+0000","last_unstale":"2023-09-18T20:45:09.542510+0000","last_undegraded":"2023-09-18T20:45:09.542510+0000","last_fullsized":"2023-09-18T20:45:09.542510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5270","last_scrub_stamp":"2023-09-18T20:45:09.542437+0000","last_deep_scrub":"32'5270","last_deep_scrub_stamp":"2023-09-18T20:45:09.542437+0000","last_clean_scrub_stamp":"2023-09-18T20:45:09.542437+0000","objects_scrubbed":32,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T06:17:34.240394+0000","scrub_duration":0.37833424599999999,"objects_trimmed":0,"snaptrim_duration":0.00065078900000000003,"stat_sum":{"num_bytes":130023440,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5151,"num_read_kb":20556,"num_write":5196,"num_write_kb":147613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"32'3845","reported_seq":7637,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.258030+0000","last_change":"2023-09-18T20:45:05.985307+0000","last_active":"2023-09-18T20:45:33.258030+0000","last_peered":"2023-09-18T20:45:33.258030+0000","last_clean":"2023-09-18T20:45:33.258030+0000","last_became_active":"2023-09-18T20:40:38.783864+0000","last_became_peered":"2023-09-18T20:40:38.783864+0000","last_unstale":"2023-09-18T20:45:33.258030+0000","last_undegraded":"2023-09-18T20:45:33.258030+0000","last_fullsized":"2023-09-18T20:45:33.258030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3845","last_scrub_stamp":"2023-09-18T20:45:05.985140+0000","last_deep_scrub":"32'3845","last_deep_scrub_stamp":"2023-09-18T20:45:05.985140+0000","last_clean_scrub_stamp":"2023-09-18T20:45:05.985140+0000","objects_scrubbed":22,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:28:59.141917+0000","scrub_duration":0.24710554500000001,"objects_trimmed":0,"snaptrim_duration":0.00038663499999999997,"stat_sum":{"num_bytes":88080384,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3944,"num_read_kb":14556,"num_write":3569,"num_write_kb":100159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"32'7193","reported_seq":14069,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.255116+0000","last_change":"2023-09-18T20:45:05.293657+0000","last_active":"2023-09-18T20:45:33.255116+0000","last_peered":"2023-09-18T20:45:33.255116+0000","last_clean":"2023-09-18T20:45:33.255116+0000","last_became_active":"2023-09-18T20:40:38.788920+0000","last_became_peered":"2023-09-18T20:40:38.788920+0000","last_unstale":"2023-09-18T20:45:33.255116+0000","last_undegraded":"2023-09-18T20:45:33.255116+0000","last_fullsized":"2023-09-18T20:45:33.255116+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'7193","last_scrub_stamp":"2023-09-18T20:45:05.293502+0000","last_deep_scrub":"32'7193","last_deep_scrub_stamp":"2023-09-18T20:45:05.293502+0000","last_clean_scrub_stamp":"2023-09-18T20:45:05.293502+0000","objects_scrubbed":44,"log_size":7193,"log_dups_size":0,"ondisk_log_size":7193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:47:04.282723+0000","scrub_duration":0.55523153300000005,"objects_trimmed":0,"snaptrim_duration":0.000746088,"stat_sum":{"num_bytes":171089920,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":7114,"num_read_kb":27284,"num_write":6867,"num_write_kb":191659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":507,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"32'3602","reported_seq":7157,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:22.427211+0000","last_change":"2023-09-18T20:45:22.427306+0000","last_active":"2023-09-18T20:45:22.427211+0000","last_peered":"2023-09-18T20:45:22.427211+0000","last_clean":"2023-09-18T20:45:22.427211+0000","last_became_active":"2023-09-18T20:40:38.788781+0000","last_became_peered":"2023-09-18T20:40:38.788781+0000","last_unstale":"2023-09-18T20:45:22.427211+0000","last_undegraded":"2023-09-18T20:45:22.427211+0000","last_fullsized":"2023-09-18T20:45:22.427211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3602","last_scrub_stamp":"2023-09-18T20:45:22.427121+0000","last_deep_scrub":"32'3602","last_deep_scrub_stamp":"2023-09-18T20:45:22.427121+0000","last_clean_scrub_stamp":"2023-09-18T20:45:22.427121+0000","objects_scrubbed":21,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T08:28:06.252173+0000","scrub_duration":0.25336082500000001,"objects_trimmed":0,"snaptrim_duration":0.0013806230000000001,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3505,"num_read_kb":14020,"num_write":3554,"num_write_kb":100116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"32'4666","reported_seq":9117,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:36.290132+0000","last_change":"2023-09-18T20:45:19.007493+0000","last_active":"2023-09-18T20:45:36.290132+0000","last_peered":"2023-09-18T20:45:36.290132+0000","last_clean":"2023-09-18T20:45:36.290132+0000","last_became_active":"2023-09-18T20:40:38.786061+0000","last_became_peered":"2023-09-18T20:40:38.786061+0000","last_unstale":"2023-09-18T20:45:36.290132+0000","last_undegraded":"2023-09-18T20:45:36.290132+0000","last_fullsized":"2023-09-18T20:45:36.290132+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4663","last_scrub_stamp":"2023-09-18T20:45:19.007285+0000","last_deep_scrub":"32'4663","last_deep_scrub_stamp":"2023-09-18T20:45:19.007285+0000","last_clean_scrub_stamp":"2023-09-18T20:45:19.007285+0000","objects_scrubbed":28,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T20:52:31.949093+0000","scrub_duration":0.34925243299999997,"objects_trimmed":0,"snaptrim_duration":0.00078825100000000001,"stat_sum":{"num_bytes":108105728,"num_objects":29,"num_object_clones":1,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4602,"num_read_kb":17631,"num_write":4375,"num_write_kb":120507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"32'4140","reported_seq":8377,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:11.573695+0000","last_change":"2023-09-18T20:45:11.573695+0000","last_active":"2023-09-18T20:45:11.573695+0000","last_peered":"2023-09-18T20:45:11.573695+0000","last_clean":"2023-09-18T20:45:11.573695+0000","last_became_active":"2023-09-18T20:40:38.787925+0000","last_became_peered":"2023-09-18T20:40:38.787925+0000","last_unstale":"2023-09-18T20:45:11.573695+0000","last_undegraded":"2023-09-18T20:45:11.573695+0000","last_fullsized":"2023-09-18T20:45:11.573695+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4140","last_scrub_stamp":"2023-09-18T20:45:11.573556+0000","last_deep_scrub":"32'4140","last_deep_scrub_stamp":"2023-09-18T20:45:11.573556+0000","last_clean_scrub_stamp":"2023-09-18T20:45:11.573556+0000","objects_scrubbed":26,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:21:44.192203+0000","scrub_duration":0.306416574,"objects_trimmed":0,"snaptrim_duration":0.0016224029999999999,"stat_sum":{"num_bytes":105390080,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4188,"num_read_kb":16752,"num_write":4077,"num_write_kb":118568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"32'6121","reported_seq":12240,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:14.708033+0000","last_change":"2023-09-18T20:45:14.708099+0000","last_active":"2023-09-18T20:45:14.708033+0000","last_peered":"2023-09-18T20:45:14.708033+0000","last_clean":"2023-09-18T20:45:14.708033+0000","last_became_active":"2023-09-18T20:40:38.789792+0000","last_became_peered":"2023-09-18T20:40:38.789792+0000","last_unstale":"2023-09-18T20:45:14.708033+0000","last_undegraded":"2023-09-18T20:45:14.708033+0000","last_fullsized":"2023-09-18T20:45:14.708033+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6121","last_scrub_stamp":"2023-09-18T20:45:14.707953+0000","last_deep_scrub":"32'6121","last_deep_scrub_stamp":"2023-09-18T20:45:14.707953+0000","last_clean_scrub_stamp":"2023-09-18T20:45:14.707953+0000","objects_scrubbed":37,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:28:20.542024+0000","scrub_duration":0.45992617699999999,"objects_trimmed":0,"snaptrim_duration":0.0013474559999999999,"stat_sum":{"num_bytes":150994994,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6069,"num_read_kb":24276,"num_write":6033,"num_write_kb":171405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"32'4189","reported_seq":8256,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.256526+0000","last_change":"2023-09-18T20:45:22.941463+0000","last_active":"2023-09-18T20:45:33.256526+0000","last_peered":"2023-09-18T20:45:33.256526+0000","last_clean":"2023-09-18T20:45:33.256526+0000","last_became_active":"2023-09-18T20:40:38.786591+0000","last_became_peered":"2023-09-18T20:40:38.786591+0000","last_unstale":"2023-09-18T20:45:33.256526+0000","last_undegraded":"2023-09-18T20:45:33.256526+0000","last_fullsized":"2023-09-18T20:45:33.256526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4189","last_scrub_stamp":"2023-09-18T20:45:22.941307+0000","last_deep_scrub":"32'4189","last_deep_scrub_stamp":"2023-09-18T20:45:22.941307+0000","last_clean_scrub_stamp":"2023-09-18T20:45:22.941307+0000","objects_scrubbed":27,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T03:25:26.878410+0000","scrub_duration":0.29085809800000001,"objects_trimmed":0,"snaptrim_duration":0.00046202999999999999,"stat_sum":{"num_bytes":97095696,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4224,"num_read_kb":15634,"num_write":3896,"num_write_kb":109676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"32'4556","reported_seq":9084,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:40.630114+0000","last_change":"2023-09-18T20:45:40.630166+0000","last_active":"2023-09-18T20:45:40.630114+0000","last_peered":"2023-09-18T20:45:40.630114+0000","last_clean":"2023-09-18T20:45:40.630114+0000","last_became_active":"2023-09-18T20:40:38.788169+0000","last_became_peered":"2023-09-18T20:40:38.788169+0000","last_unstale":"2023-09-18T20:45:40.630114+0000","last_undegraded":"2023-09-18T20:45:40.630114+0000","last_fullsized":"2023-09-18T20:45:40.630114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4556","last_scrub_stamp":"2023-09-18T20:45:40.630025+0000","last_deep_scrub":"32'4556","last_deep_scrub_stamp":"2023-09-18T20:45:40.630025+0000","last_clean_scrub_stamp":"2023-09-18T20:45:40.630025+0000","objects_scrubbed":29,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T23:25:18.345329+0000","scrub_duration":0.33102039799999999,"objects_trimmed":0,"snaptrim_duration":0.00086856900000000003,"stat_sum":{"num_bytes":113254450,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":4479,"num_read_kb":17916,"num_write":4483,"num_write_kb":128393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"32'6284","reported_seq":12373,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:21.654952+0000","last_change":"2023-09-18T20:45:21.655012+0000","last_active":"2023-09-18T20:45:21.654952+0000","last_peered":"2023-09-18T20:45:21.654952+0000","last_clean":"2023-09-18T20:45:21.654952+0000","last_became_active":"2023-09-18T20:40:38.783749+0000","last_became_peered":"2023-09-18T20:40:38.783749+0000","last_unstale":"2023-09-18T20:45:21.654952+0000","last_undegraded":"2023-09-18T20:45:21.654952+0000","last_fullsized":"2023-09-18T20:45:21.654952+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6284","last_scrub_stamp":"2023-09-18T20:45:21.654864+0000","last_deep_scrub":"32'6284","last_deep_scrub_stamp":"2023-09-18T20:45:21.654864+0000","last_clean_scrub_stamp":"2023-09-18T20:45:21.654864+0000","objects_scrubbed":37,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:26:00.002975+0000","scrub_duration":0.44780350499999999,"objects_trimmed":0,"snaptrim_duration":0.00085165100000000004,"stat_sum":{"num_bytes":151543808,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":6039,"num_read_kb":24156,"num_write":6201,"num_write_kb":172296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"32'5004","reported_seq":9983,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.203665+0000","last_change":"2023-09-18T20:45:13.065201+0000","last_active":"2023-09-18T20:45:33.203665+0000","last_peered":"2023-09-18T20:45:33.203665+0000","last_clean":"2023-09-18T20:45:33.203665+0000","last_became_active":"2023-09-18T20:40:38.788586+0000","last_became_peered":"2023-09-18T20:40:38.788586+0000","last_unstale":"2023-09-18T20:45:33.203665+0000","last_undegraded":"2023-09-18T20:45:33.203665+0000","last_fullsized":"2023-09-18T20:45:33.203665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5004","last_scrub_stamp":"2023-09-18T20:45:13.065001+0000","last_deep_scrub":"32'5004","last_deep_scrub_stamp":"2023-09-18T20:45:13.065001+0000","last_clean_scrub_stamp":"2023-09-18T20:45:13.065001+0000","objects_scrubbed":33,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:18:42.417094+0000","scrub_duration":0.371655493,"objects_trimmed":0,"snaptrim_duration":0.00057542900000000002,"stat_sum":{"num_bytes":121643108,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4929,"num_read_kb":19686,"num_write":4935,"num_write_kb":138349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":340,"num_omap_keys":5,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'4608","reported_seq":9080,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.797062+0000","last_change":"2023-09-18T20:44:49.462443+0000","last_active":"2023-09-18T20:45:00.797062+0000","last_peered":"2023-09-18T20:45:00.797062+0000","last_clean":"2023-09-18T20:45:00.797062+0000","last_became_active":"2023-09-18T20:40:38.788411+0000","last_became_peered":"2023-09-18T20:40:38.788411+0000","last_unstale":"2023-09-18T20:45:00.797062+0000","last_undegraded":"2023-09-18T20:45:00.797062+0000","last_fullsized":"2023-09-18T20:45:00.797062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4608","last_scrub_stamp":"2023-09-18T20:44:49.462306+0000","last_deep_scrub":"31'4608","last_deep_scrub_stamp":"2023-09-18T20:44:49.462306+0000","last_clean_scrub_stamp":"2023-09-18T20:44:49.462306+0000","objects_scrubbed":27,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T00:54:51.936441+0000","scrub_duration":0.337859574,"objects_trimmed":0,"snaptrim_duration":0.001418414,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4422,"num_read_kb":17688,"num_write":4538,"num_write_kb":128580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"32'4458","reported_seq":8876,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:18.595236+0000","last_change":"2023-09-18T20:45:18.595236+0000","last_active":"2023-09-18T20:45:18.595236+0000","last_peered":"2023-09-18T20:45:18.595236+0000","last_clean":"2023-09-18T20:45:18.595236+0000","last_became_active":"2023-09-18T20:40:38.784301+0000","last_became_peered":"2023-09-18T20:40:38.784301+0000","last_unstale":"2023-09-18T20:45:18.595236+0000","last_undegraded":"2023-09-18T20:45:18.595236+0000","last_fullsized":"2023-09-18T20:45:18.595236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4458","last_scrub_stamp":"2023-09-18T20:45:18.595102+0000","last_deep_scrub":"32'4458","last_deep_scrub_stamp":"2023-09-18T20:45:18.595102+0000","last_clean_scrub_stamp":"2023-09-18T20:45:18.595102+0000","objects_scrubbed":27,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:43:59.143459+0000","scrub_duration":0.39233341900000002,"objects_trimmed":0,"snaptrim_duration":0.0014517309999999999,"stat_sum":{"num_bytes":109060096,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":4368,"num_read_kb":17472,"num_write":4395,"num_write_kb":123940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"32'5131","reported_seq":10376,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:22.060238+0000","last_change":"2023-09-18T20:45:22.060238+0000","last_active":"2023-09-18T20:45:22.060238+0000","last_peered":"2023-09-18T20:45:22.060238+0000","last_clean":"2023-09-18T20:45:22.060238+0000","last_became_active":"2023-09-18T20:40:38.785955+0000","last_became_peered":"2023-09-18T20:40:38.785955+0000","last_unstale":"2023-09-18T20:45:22.060238+0000","last_undegraded":"2023-09-18T20:45:22.060238+0000","last_fullsized":"2023-09-18T20:45:22.060238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5131","last_scrub_stamp":"2023-09-18T20:45:22.060098+0000","last_deep_scrub":"32'5131","last_deep_scrub_stamp":"2023-09-18T20:45:22.060098+0000","last_clean_scrub_stamp":"2023-09-18T20:45:22.060098+0000","objects_scrubbed":32,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T04:04:56.690721+0000","scrub_duration":0.37624659799999999,"objects_trimmed":0,"snaptrim_duration":0.00056327000000000005,"stat_sum":{"num_bytes":130031616,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":5195,"num_read_kb":20780,"num_write":5057,"num_write_kb":147044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"32'6125","reported_seq":11985,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:18.801745+0000","last_change":"2023-09-18T20:45:18.801745+0000","last_active":"2023-09-18T20:45:18.801745+0000","last_peered":"2023-09-18T20:45:18.801745+0000","last_clean":"2023-09-18T20:45:18.801745+0000","last_became_active":"2023-09-18T20:40:38.785056+0000","last_became_peered":"2023-09-18T20:40:38.785056+0000","last_unstale":"2023-09-18T20:45:18.801745+0000","last_undegraded":"2023-09-18T20:45:18.801745+0000","last_fullsized":"2023-09-18T20:45:18.801745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6125","last_scrub_stamp":"2023-09-18T20:45:18.801592+0000","last_deep_scrub":"32'6125","last_deep_scrub_stamp":"2023-09-18T20:45:18.801592+0000","last_clean_scrub_stamp":"2023-09-18T20:45:18.801592+0000","objects_scrubbed":36,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:41:37.387977+0000","scrub_duration":0.56301445699999997,"objects_trimmed":0,"snaptrim_duration":0.001234221,"stat_sum":{"num_bytes":146808832,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5810,"num_read_kb":23240,"num_write":6046,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"32'6972","reported_seq":13764,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.259317+0000","last_change":"2023-09-18T20:45:24.166522+0000","last_active":"2023-09-18T20:45:33.259317+0000","last_peered":"2023-09-18T20:45:33.259317+0000","last_clean":"2023-09-18T20:45:33.259317+0000","last_became_active":"2023-09-18T20:40:38.786933+0000","last_became_peered":"2023-09-18T20:40:38.786933+0000","last_unstale":"2023-09-18T20:45:33.259317+0000","last_undegraded":"2023-09-18T20:45:33.259317+0000","last_fullsized":"2023-09-18T20:45:33.259317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6972","last_scrub_stamp":"2023-09-18T20:45:24.166386+0000","last_deep_scrub":"32'6972","last_deep_scrub_stamp":"2023-09-18T20:45:24.166386+0000","last_clean_scrub_stamp":"2023-09-18T20:45:24.166386+0000","objects_scrubbed":42,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T02:47:07.763983+0000","scrub_duration":0.52516249199999998,"objects_trimmed":0,"snaptrim_duration":0.00059420000000000002,"stat_sum":{"num_bytes":164110351,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6953,"num_read_kb":26506,"num_write":6659,"num_write_kb":186156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"32'6714","reported_seq":13454,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:00.809187+0000","last_change":"2023-09-18T20:44:52.726361+0000","last_active":"2023-09-18T20:45:00.809187+0000","last_peered":"2023-09-18T20:45:00.809187+0000","last_clean":"2023-09-18T20:45:00.809187+0000","last_became_active":"2023-09-18T20:40:38.785545+0000","last_became_peered":"2023-09-18T20:40:38.785545+0000","last_unstale":"2023-09-18T20:45:00.809187+0000","last_undegraded":"2023-09-18T20:45:00.809187+0000","last_fullsized":"2023-09-18T20:45:00.809187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6713","last_scrub_stamp":"2023-09-18T20:44:52.726150+0000","last_deep_scrub":"31'6713","last_deep_scrub_stamp":"2023-09-18T20:44:52.726150+0000","last_clean_scrub_stamp":"2023-09-18T20:44:52.726150+0000","objects_scrubbed":41,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T01:53:09.298051+0000","scrub_duration":0.52103732999999997,"objects_trimmed":0,"snaptrim_duration":0.00066659399999999995,"stat_sum":{"num_bytes":168390706,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":6690,"num_read_kb":26760,"num_write":6623,"num_write_kb":190145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"32'5405","reported_seq":10504,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:36.746038+0000","last_change":"2023-09-18T20:45:36.746096+0000","last_active":"2023-09-18T20:45:36.746038+0000","last_peered":"2023-09-18T20:45:36.746038+0000","last_clean":"2023-09-18T20:45:36.746038+0000","last_became_active":"2023-09-18T20:40:38.785077+0000","last_became_peered":"2023-09-18T20:40:38.785077+0000","last_unstale":"2023-09-18T20:45:36.746038+0000","last_undegraded":"2023-09-18T20:45:36.746038+0000","last_fullsized":"2023-09-18T20:45:36.746038+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'5405","last_scrub_stamp":"2023-09-18T20:45:36.745966+0000","last_deep_scrub":"32'5405","last_deep_scrub_stamp":"2023-09-18T20:45:36.745966+0000","last_clean_scrub_stamp":"2023-09-18T20:45:36.745966+0000","objects_scrubbed":33,"log_size":5405,"log_dups_size":0,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-20T05:11:07.940907+0000","scrub_duration":0.39702229900000002,"objects_trimmed":0,"snaptrim_duration":0.00072078500000000005,"stat_sum":{"num_bytes":130564096,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5049,"num_read_kb":20196,"num_write":5320,"num_write_kb":148316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"32'4697","reported_seq":9422,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-09-18T20:45:33.573808+0000","last_change":"2023-09-18T20:45:33.573808+0000","last_active":"2023-09-18T20:45:33.573808+0000","last_peered":"2023-09-18T20:45:33.573808+0000","last_clean":"2023-09-18T20:45:33.573808+0000","last_became_active":"2023-09-18T20:40:38.785373+0000","last_became_peered":"2023-09-18T20:40:38.785373+0000","last_unstale":"2023-09-18T20:45:33.573808+0000","last_undegraded":"2023-09-18T20:45:33.573808+0000","last_fullsized":"2023-09-18T20:45:33.573808+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'4697","last_scrub_stamp":"2023-09-18T20:45:33.573595+0000","last_deep_scrub":"32'4697","last_deep_scrub_stamp":"2023-09-18T20:45:33.573595+0000","last_clean_scrub_stamp":"2023-09-18T20:45:33.573595+0000","objects_scrubbed":30,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-19T21:28:42.151871+0000","scrub_duration":0.34800658800000001,"objects_trimmed":0,"snaptrim_duration":0.0011797999999999999,"stat_sum":{"num_bytes":116555795,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4675,"num_read_kb":18915,"num_write":4620,"num_write_kb":129498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4216345174,"num_objects":1061,"num_object_clones":2,"num_object_copies":2122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":169718,"num_read_kb":673327,"num_write":169077,"num_write_kb":4770873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3462,"num_omap_keys":78,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8392720384,"data_stored":8402551980,"data_compressed":381586,"data_compressed_allocated":2195456,"data_compressed_original":11812864,"omap_allocated":0,"internal_metadata":0},"log_size":172631,"ondisk_log_size":172631,"up":64,"acting":64,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11524,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738449,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2820280,"kb_used_data":2397808,"kb_used_omap":1,"kb_used_meta":422462,"kb_avail":91551560,"statfs":{"total":96636764160,"available":93748797440,"internally_reserved":0,"allocated":2455355392,"data_stored":2456650470,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":1594,"internal_metadata":432601542},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:45:29 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":1.071,"15min":0.55300000000000005},"min":{"1min":0.16,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.54200000000000004,"5min":38.908999999999999,"15min":38.908999999999999},"last":0.184},{"interface":"front","average":{"1min":0.311,"5min":1.042,"15min":0.55000000000000004},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":38.030000000000001,"15min":38.030000000000001},"last":0.20899999999999999}]},{"osd":1,"last update":"Mon Sep 18 20:45:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":1.331,"15min":0.64600000000000002},"min":{"1min":0.216,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45100000000000001,"5min":73.347999999999999,"15min":73.347999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":1.3280000000000001,"15min":0.65600000000000003},"min":{"1min":0.245,"5min":0.19,"15min":0.19},"max":{"1min":0.51900000000000002,"5min":76.748000000000005,"15min":76.748000000000005},"last":0.29999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2978392,"kb_used_data":2532304,"kb_used_omap":1,"kb_used_meta":446078,"kb_avail":91393448,"statfs":{"total":96636764160,"available":93586890752,"internally_reserved":0,"allocated":2593079296,"data_stored":2596910388,"data_compressed":88581,"data_compressed_allocated":434176,"data_compressed_original":4304896,"omap_allocated":1594,"internal_metadata":456784326},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 18 20:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.46600000000000003,"15min":0.34799999999999998},"min":{"1min":0.20100000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.38400000000000001,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.59799999999999998,"15min":0.40600000000000003},"min":{"1min":0.21299999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.45800000000000002,"5min":3.585,"15min":3.585},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Sep 18 20:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.60999999999999999,"15min":0.39400000000000002},"min":{"1min":0.216,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.41599999999999998,"5min":12.382,"15min":12.382},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.53700000000000003,"15min":0.40200000000000002},"min":{"1min":0.23000000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.45700000000000002,"5min":3.625,"15min":3.625},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3833664,"kb_used_data":3266808,"kb_used_omap":1,"kb_used_meta":566846,"kb_avail":90538176,"statfs":{"total":96636764160,"available":92711092224,"internally_reserved":0,"allocated":3345211392,"data_stored":3350480231,"data_compressed":194816,"data_compressed_allocated":1122304,"data_compressed_original":6443008,"omap_allocated":1594,"internal_metadata":580450758},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 18 20:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.81699999999999995,"15min":0.46000000000000002},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.41599999999999998,"5min":23.518999999999998,"15min":23.518999999999998},"last":0.441},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.72599999999999998,"15min":0.42999999999999999},"min":{"1min":0.223,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.45800000000000002,"5min":11.029999999999999,"15min":11.029999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Sep 18 20:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.94899999999999995,"15min":0.505},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.48899999999999999,"5min":23.952999999999999,"15min":23.952999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.33900000000000002,"5min":1.145,"15min":0.58599999999999997},"min":{"1min":0.22500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.47499999999999998,"5min":32.715000000000003,"15min":32.715000000000003},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5762,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592751616,"data_stored":2596217229,"data_compressed":82819,"data_compressed_allocated":397312,"data_compressed_original":3731456,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3344883712,"data_stored":3349787072,"data_compressed":189054,"data_compressed_allocated":1085440,"data_compressed_original":5869568,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2455085056,"data_stored":2456547679,"data_compressed":109713,"data_compressed_allocated":712704,"data_compressed_original":2211840,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-18T20:45:46.503 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-18T20:45:46.839 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-18T20:45:46.840 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-18T20:45:46.840 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-18T20:45:46.841 INFO:teuthology.orchestra.run:waiting for 300 2023-09-18T20:45:46.905 INFO:tasks.ceph.osd.0:Stopped 2023-09-18T20:45:46.905 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-18T20:45:46.906 INFO:teuthology.orchestra.run:waiting for 300 2023-09-18T20:45:49.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-09-18T20:45:53.008 INFO:tasks.ceph.osd.1:Stopped 2023-09-18T20:45:53.009 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-18T20:45:53.009 INFO:teuthology.orchestra.run:waiting for 300 2023-09-18T20:45:55.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-18T20:45:55.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-09-18T20:45:59.081 INFO:tasks.ceph.osd.2:Stopped 2023-09-18T20:45:59.082 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-18T20:45:59.082 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-18T20:45:59.083 INFO:teuthology.orchestra.run:waiting for 300 2023-09-18T20:45:59.112 INFO:tasks.ceph.mgr.x:Stopped 2023-09-18T20:45:59.112 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-18T20:45:59.112 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-18T20:45:59.113 INFO:teuthology.orchestra.run:waiting for 300 2023-09-18T20:45:59.171 INFO:tasks.ceph.mon.a:Stopped 2023-09-18T20:45:59.172 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-18T20:45:59.172 DEBUG:teuthology.orchestra.run.smithi149:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-09-18T20:45:59.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:45:59.261 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-18T20:45:59.599 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:45:59.600 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-18T20:45:59.745 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:45:59.746 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-18T20:45:59.841 INFO:tasks.ceph:Archiving mon data... 2023-09-18T20:45:59.843 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734/data/mon.a.tgz 2023-09-18T20:45:59.843 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2023-09-18T20:45:59.876 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.esD3OpypmS 2023-09-18T20:45:59.877 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.esD3OpypmS 2023-09-18T20:46:00.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-18T20:46:00.037 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.esD3OpypmS 2023-09-18T20:46:00.170 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.esD3OpypmS is 336KB 2023-09-18T20:46:00.265 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.esD3OpypmS 2023-09-18T20:46:00.282 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-18T20:46:00.282 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-18T20:46:00.348 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-18T20:46:00.348 DEBUG:teuthology.orchestra.run.smithi149:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-18T20:46:00.433 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-18T20:46:00.438 INFO:tasks.ceph:Archiving crash dumps... 2023-09-18T20:46:00.440 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734/remote/smithi149/crash 2023-09-18T20:46:00.442 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-18T20:46:00.467 INFO:tasks.ceph:Compressing logs... 2023-09-18T20:46:00.468 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-18T20:46:00.583 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105945.log 2023-09-18T20:46:00.583 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104398.log 2023-09-18T20:46:00.584 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106060.log 2023-09-18T20:46:00.584 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105945.log.gz 2023-09-18T20:46:00.584 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106319.log 2023-09-18T20:46:00.584 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104398.log.gz 2023-09-18T20:46:00.585 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106060.log.gz 2023-09-18T20:46:00.585 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-09-18T20:46:00.585 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106400.log 2023-09-18T20:46:00.586 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106319.log.gz 2023-09-18T20:46:00.586 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-09-18T20:46:00.586 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106007.log 2023-09-18T20:46:00.586 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106400.log.gz 2023-09-18T20:46:00.587 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98536.log 2023-09-18T20:46:00.587 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106252.log 2023-09-18T20:46:00.587 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106007.log.gz 2023-09-18T20:46:00.588 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.98536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106142.log 2023-09-18T20:46:00.588 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98536.log.gz 2023-09-18T20:46:00.588 INFO:teuthology.orchestra.run.smithi149.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-09-18T20:46:00.589 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104669.log 2023-09-18T20:46:00.589 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106252.log.gz 2023-09-18T20:46:00.589 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106142.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-09-18T20:46:00.589 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106142.log.gz 2023-09-18T20:46:00.590 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103108.log 2023-09-18T20:46:00.590 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104669.log.gz 2023-09-18T20:46:00.590 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104125.log 2023-09-18T20:46:00.591 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 /var/log/ceph/ceph-client.admin.103108.log: --verbose -- /var/log/ceph/ceph-client.admin.104142.log 2023-09-18T20:46:00.591 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103108.log.gz 2023-09-18T20:46:00.591 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106172.log 2023-09-18T20:46:00.592 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104125.log.gz 2023-09-18T20:46:00.592 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105891.log 2023-09-18T20:46:00.592 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104142.log.gz 2023-09-18T20:46:00.593 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106172.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.102786.log 2023-09-18T20:46:00.593 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106172.log.gz 2023-09-18T20:46:00.593 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105604.log 2023-09-18T20:46:00.593 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105891.log.gz 2023-09-18T20:46:00.594 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102418.log 2023-09-18T20:46:00.594 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102786.log.gz 2023-09-18T20:46:00.594 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105520.log 2023-09-18T20:46:00.595 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105604.log.gz 2023-09-18T20:46:00.595 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106722.log 2023-09-18T20:46:00.595 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102418.log.gz 2023-09-18T20:46:00.596 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105492.log 2023-09-18T20:46:00.596 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105520.log.gz 2023-09-18T20:46:00.596 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104324.log 2023-09-18T20:46:00.596 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106722.log.gz 2023-09-18T20:46:00.597 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106446.log 2023-09-18T20:46:00.597 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105492.log.gz 2023-09-18T20:46:00.597 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102692.log 2023-09-18T20:46:00.597 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104324.log.gz 2023-09-18T20:46:00.598 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106446.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.106446.log.gz--verbose 2023-09-18T20:46:00.598 INFO:teuthology.orchestra.run.smithi149.stderr:-- /var/log/ceph/ceph-client.admin.105565.log 2023-09-18T20:46:00.598 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106234.log 2023-09-18T20:46:00.598 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102692.log.gz 2023-09-18T20:46:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106906.log 2023-09-18T20:46:00.599 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105565.log.gz 2023-09-18T20:46:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106234.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.106234.log.gz 2023-09-18T20:46:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.104270.log 2023-09-18T20:46:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104361.log 2023-09-18T20:46:00.600 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106906.log.gz 2023-09-18T20:46:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106301.log 2023-09-18T20:46:00.600 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104270.log.gz 2023-09-18T20:46:00.601 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104344.log 2023-09-18T20:46:00.601 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104361.log.gz 2023-09-18T20:46:00.601 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102510.log 2023-09-18T20:46:00.601 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106301.log.gz 2023-09-18T20:46:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102694.log 2023-09-18T20:46:00.602 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104344.log.gz 2023-09-18T20:46:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102510.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.102510.log.gz/var/log/ceph/ceph-client.admin.102556.log 2023-09-18T20:46:00.602 INFO:teuthology.orchestra.run.smithi149.stderr: 2023-09-18T20:46:00.603 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98983.log 2023-09-18T20:46:00.603 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102694.log.gz 2023-09-18T20:46:00.603 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105779.log 2023-09-18T20:46:00.603 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102556.log.gz 2023-09-18T20:46:00.604 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.98983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98983.log.gz 2023-09-18T20:46:00.604 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104179.log 2023-09-18T20:46:00.604 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105976.log 2023-09-18T20:46:00.604 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105779.log.gz 2023-09-18T20:46:00.605 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104476.log 2023-09-18T20:46:00.605 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104179.log.gz 2023-09-18T20:46:00.605 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104063.log 2023-09-18T20:46:00.605 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105976.log.gz 2023-09-18T20:46:00.606 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106814.log 2023-09-18T20:46:00.606 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104476.log.gz 2023-09-18T20:46:00.606 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103154.log 2023-09-18T20:46:00.607 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104063.log.gz 2023-09-18T20:46:00.607 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106860.log 2023-09-18T20:46:00.607 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106814.log.gz 2023-09-18T20:46:00.608 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.103154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102259.log 2023-09-18T20:46:00.608 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103154.log.gz 2023-09-18T20:46:00.608 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104378.log 2023-09-18T20:46:00.608 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106860.log.gz 2023-09-18T20:46:00.609 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102259.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-09-18T20:46:00.609 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102259.log.gz 2023-09-18T20:46:00.609 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98755.log 2023-09-18T20:46:00.609 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104378.log.gz 2023-09-18T20:46:00.610 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104199.log 2023-09-18T20:46:00.610 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.98755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106190.log 2023-09-18T20:46:00.610 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98755.log.gz 2023-09-18T20:46:00.610 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103200.log 2023-09-18T20:46:00.611 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104199.log.gz 2023-09-18T20:46:00.611 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106284.log 2023-09-18T20:46:00.611 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106190.log.gz 2023-09-18T20:46:00.611 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.103200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102602.log 2023-09-18T20:46:00.612 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103200.log.gz 2023-09-18T20:46:00.612 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106337.log 2023-09-18T20:46:00.612 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106284.log.gz 2023-09-18T20:46:00.612 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105863.log 2023-09-18T20:46:00.613 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102602.log.gz 2023-09-18T20:46:00.613 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106337.log.gz 2023-09-18T20:46:00.613 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106676.log 2023-09-18T20:46:00.614 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106960.log 2023-09-18T20:46:00.614 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105863.log.gz 2023-09-18T20:46:00.614 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102970.log 2023-09-18T20:46:00.614 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106676.log.gz 2023-09-18T20:46:00.615 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106960.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.94100.log 2023-09-18T20:46:00.615 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106960.log.gz 2023-09-18T20:46:00.615 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102970.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.102970.log.gz 2023-09-18T20:46:00.616 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102969.log 2023-09-18T20:46:00.616 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph.tmp-client.admin.94100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104290.log 2023-09-18T20:46:00.616 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.94100.log.gz 2023-09-18T20:46:00.617 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106768.log 2023-09-18T20:46:00.617 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102969.log.gz 2023-09-18T20:46:00.617 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104290.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-09-18T20:46:00.618 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104290.log.gz 2023-09-18T20:46:00.618 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106768.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.105748.log 2023-09-18T20:46:00.618 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106768.log.gz 2023-09-18T20:46:00.618 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105717.log 2023-09-18T20:46:00.619 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105748.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-09-18T20:46:00.619 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105748.log.gz 2023-09-18T20:46:00.619 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102693.log 2023-09-18T20:46:00.620 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105717.log.gz 2023-09-18T20:46:00.620 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104307.log 2023-09-18T20:46:00.620 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106630.log 2023-09-18T20:46:00.621 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102693.log.gz 2023-09-18T20:46:00.621 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106492.log 2023-09-18T20:46:00.622 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104307.log.gz 2023-09-18T20:46:00.622 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104417.log 2023-09-18T20:46:00.622 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106630.log.gz 2023-09-18T20:46:00.623 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106492.log.gz 2023-09-18T20:46:00.623 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105917.log 2023-09-18T20:46:00.623 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104108.log 2023-09-18T20:46:00.624 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104417.log.gz 2023-09-18T20:46:00.624 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104253.log 2023-09-18T20:46:00.624 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105917.log.gz 2023-09-18T20:46:00.625 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106088.log 2023-09-18T20:46:00.625 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104108.log.gz 2023-09-18T20:46:00.625 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104617.log 2023-09-18T20:46:00.625 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104253.log.gz 2023-09-18T20:46:00.626 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105658.log 2023-09-18T20:46:00.626 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106088.log.gz 2023-09-18T20:46:00.626 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106584.log 2023-09-18T20:46:00.627 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104617.log.gz 2023-09-18T20:46:00.627 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98916.log 2023-09-18T20:46:00.627 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105658.log.gz 2023-09-18T20:46:00.628 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106538.log 2023-09-18T20:46:00.628 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106584.log.gz 2023-09-18T20:46:00.628 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102924.log 2023-09-18T20:46:00.628 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.98916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98916.log.gz 2023-09-18T20:46:00.629 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104523.log 2023-09-18T20:46:00.629 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106538.log.gz 2023-09-18T20:46:00.629 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102924.log: gzip 0.0%-5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.102924.log.gz/var/log/ceph/ceph-client.admin.106208.log 2023-09-18T20:46:00.629 INFO:teuthology.orchestra.run.smithi149.stderr: 2023-09-18T20:46:00.630 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105632.log 2023-09-18T20:46:00.630 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104523.log.gz 2023-09-18T20:46:00.630 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104775.log 2023-09-18T20:46:00.630 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106208.log.gz 2023-09-18T20:46:00.631 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105632.log.gz 2023-09-18T20:46:00.631 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102355.log 2023-09-18T20:46:00.631 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103062.log 2023-09-18T20:46:00.632 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104775.log.gz 2023-09-18T20:46:00.632 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102355.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.102355.log.gz/var/log/ceph/ceph-client.admin.107006.log 2023-09-18T20:46:00.632 INFO:teuthology.orchestra.run.smithi149.stderr: 2023-09-18T20:46:00.632 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.103062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104236.log 2023-09-18T20:46:00.633 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103062.log.gz 2023-09-18T20:46:00.633 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.107006.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.107006.log.gz-- 2023-09-18T20:46:00.633 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104090.log 2023-09-18T20:46:00.634 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105818.log 2023-09-18T20:46:00.634 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104236.log.gz 2023-09-18T20:46:00.634 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105686.log 2023-09-18T20:46:00.634 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104090.log.gz 2023-09-18T20:46:00.635 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105818.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-09-18T20:46:00.635 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105818.log.gz 2023-09-18T20:46:00.635 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.105686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102464.log 2023-09-18T20:46:00.635 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105686.log.gz 2023-09-18T20:46:00.636 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102832.log 2023-09-18T20:46:00.636 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107050.log 2023-09-18T20:46:00.636 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102464.log.gz 2023-09-18T20:46:00.637 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106114.log 2023-09-18T20:46:00.637 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102832.log.gz 2023-09-18T20:46:00.637 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.107050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104216.log 2023-09-18T20:46:00.637 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107050.log.gz 2023-09-18T20:46:00.638 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.106114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98802.log 2023-09-18T20:46:00.638 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106114.log.gz 2023-09-18T20:46:00.638 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98708.log 2023-09-18T20:46:00.639 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104216.log.gz 2023-09-18T20:46:00.639 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.98802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102831.log 2023-09-18T20:46:00.639 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98802.log.gz 2023-09-18T20:46:00.640 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.98708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104159.log 2023-09-18T20:46:00.640 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98708.log.gz 2023-09-18T20:46:00.640 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.102831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104570.log 2023-09-18T20:46:00.640 INFO:teuthology.orchestra.run.smithi149.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102831.log.gz 2023-09-18T20:46:00.641 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104159.log.gz 2023-09-18T20:46:00.641 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.104570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104570.log.gz 2023-09-18T20:46:00.775 INFO:teuthology.orchestra.run.smithi149.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-09-18T20:46:00.897 INFO:teuthology.orchestra.run.smithi149.stderr: 88.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-09-18T20:46:00.937 INFO:teuthology.orchestra.run.smithi149.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-09-18T20:46:25.714 INFO:teuthology.orchestra.run.smithi149.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-09-18T20:46:30.411 INFO:teuthology.orchestra.run.smithi149.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-09-18T20:46:30.953 INFO:teuthology.orchestra.run.smithi149.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-09-18T20:46:30.955 INFO:teuthology.orchestra.run.smithi149.stderr: 2023-09-18T20:46:30.956 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m30.432s 2023-09-18T20:46:30.956 INFO:teuthology.orchestra.run.smithi149.stderr:user 1m2.311s 2023-09-18T20:46:30.956 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m2.605s 2023-09-18T20:46:30.956 INFO:tasks.ceph:Archiving logs... 2023-09-18T20:46:30.957 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734/remote/smithi149/log 2023-09-18T20:46:30.958 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-18T20:46:40.157 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-18T20:46:40.167 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-18T20:46:40.168 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-18T20:46:40.294 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-18T20:46:40.303 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-18T20:46:40.303 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-18T20:46:40.319 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2023-09-18T20:46:40.364 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-18T20:46:40.364 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2023-09-18T20:46:40.364 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 -453us[ -453us] +/- 71ms 2023-09-18T20:46:40.364 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -556us[ -584us] +/- 57ms 2023-09-18T20:46:40.365 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +1317us[+1289us] +/- 81ms 2023-09-18T20:46:40.365 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-18T20:46:40.366 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-18T20:46:40.375 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-18T20:46:40.375 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-18T20:46:40.384 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-18T20:46:40.549 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-18T20:46:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-18T20:46:40.580 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-18T20:46:40.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-18T20:46:40.598 INFO:teuthology.task.internal:Duration was 1235.181051 seconds 2023-09-18T20:46:40.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-18T20:46:40.607 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-18T20:46:40.608 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-18T20:46:40.704 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-18T20:46:40.745 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-18T20:46:40.746 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2023-09-18T20:46:40.746 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-18T20:46:40.768 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-18T20:46:40.768 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-18T20:46:40.848 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-18T20:46:40.848 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-18T20:46:40.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-18T20:46:40.991 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-18T20:46:40.991 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-18T20:46:41.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-18T20:46:41.038 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-18T20:46:41.100 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2023-09-18T20:46:41.145 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-18T20:46:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-18T20:46:41.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-18T20:46:41.211 INFO:teuthology.task.internal:Transferring archived files... 2023-09-18T20:46:41.212 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-09-18_20:19:35-rbd:cli-wip-ajarr-repeated-rbd-client-blocklisting-distro-default-smithi/7397734/remote/smithi149 2023-09-18T20:46:41.212 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-18T20:46:41.279 INFO:teuthology.task.internal:Removing archive directory... 2023-09-18T20:46:41.279 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-18T20:46:41.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-18T20:46:41.347 INFO:teuthology.task.internal:Not uploading archives. 2023-09-18T20:46:41.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-18T20:46:41.357 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-18T20:46:41.357 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-18T20:46:41.394 INFO:teuthology.orchestra.run.smithi149.stdout: 262320 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 18 20:46 /home/ubuntu/cephtest 2023-09-18T20:46:41.395 INFO:teuthology.orchestra.run.smithi149.stdout: 659415 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 18 20:39 /home/ubuntu/cephtest/mnt.0 2023-09-18T20:46:41.395 INFO:teuthology.orchestra.run.smithi149.stdout: 659416 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 18 20:40 /home/ubuntu/cephtest/mnt.0/client.0 2023-09-18T20:46:41.395 INFO:teuthology.orchestra.run.smithi149.stdout: 1052908 4 drwxrwxr-x 7 ubuntu ubuntu 4096 Sep 18 20:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-18T20:46:41.395 INFO:teuthology.orchestra.run.smithi149.stdout: 2 4 drwxr-xr-x 3 ubuntu root 4096 Sep 18 20:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage3 2023-09-18T20:46:41.396 INFO:teuthology.orchestra.run.smithi149.stdout: 12 819204 -rw-r--r-- 1 ubuntu ubuntu 838860800 Sep 18 20:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage3/test 2023-09-18T20:46:41.396 INFO:teuthology.orchestra.run.smithi149.stdout: 11 16 drwx------ 2 root root 16384 Sep 18 20:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage3/lost+found 2023-09-18T20:46:41.396 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage3/lost+found’: Permission denied 2023-09-18T20:46:41.397 INFO:teuthology.orchestra.run.smithi149.stdout: 2 4 drwxr-xr-x 3 ubuntu root 4096 Sep 18 20:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage1 2023-09-18T20:46:41.397 INFO:teuthology.orchestra.run.smithi149.stdout: 12 819204 -rw-r--r-- 1 ubuntu ubuntu 838860800 Sep 18 20:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage1/test 2023-09-18T20:46:41.397 INFO:teuthology.orchestra.run.smithi149.stdout: 11 16 drwx------ 2 root root 16384 Sep 18 20:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage1/lost+found 2023-09-18T20:46:41.398 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/home/ubuntu/cephtest/mnt.0/client.0/tmp/mntimage1/lost+found’: Permission denied 2023-09-19T08:26:21.323 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-09-19T08:26:21.325 DEBUG:teuthology.task.console_log:Killing console logger for smithi149 2023-09-19T08:26:21.326 DEBUG:teuthology.exit:Finished running handlers